Support tickets about a confusing settings page drop to almost nothing after a help article ships, and the same questions reappear as pre-purchase objections where nobody is counting them
The Tickets That Moved — Volume I, Understanding Humans Before Products · Systems · Investigation · Core. Support tickets about a confusing settings page drop to almost nothing after a help article ships, and the same questions reappear as pre-purchase objections where nobody is counting them
01 · Opening
Support tickets about a confusing settings page drop to almost nothing after a help article ships, and the same questions reappear as pre-purchase objections where nobody is counting them
02 · The setup
The settings page has generated the same three questions for two years. The help article takes a week and the tickets stop.
Priya: Down ninety-one percent. Support have got their Mondays back.
03 · The setup
Settings tickets per week — Before article: 34; After article: 3; Article views: 1,900; Settings page unchanged: yes
04 · The evidence
Six weeks later Sam mentions something in passing that does not fit.
Sam: Demos keep stalling on the settings screen. I am answering the same three questions all day.
05 · The evidence
Maya: Which three?
Sam: The ones in your help article. I sent it to a prospect last week.
06 · The evidence
The questions did not stop. They moved to a room where nobody logs anything.
Maya thinks: Support counts tickets. Sales counts deals. Nobody counts questions asked in a demo.
07 · The evidence
Maya asks sales to tally the three questions for a fortnight. It is the first time anyone has.
Same three questions, both sides of the sale — Support tickets, per week: 3; Asked in demos, per week: 29; Demos where it stalled the call: 11; Counted anywhere before this: no
08 · The evidence
Dev: So the article worked.
Maya: It did. It moved the cost from a place we measure to a place we do not.
09 · The evidence
Every measurement has a boundary, and the boundary is usually a team. Support counts tickets. Sales counts deals. A problem that crosses from one to the other looks like a problem being solved, because the only people counting stopped seeing it.
The help article did real work — nineteen hundred people read it and got their answer. What it did not do is make the settings page comprehensible, so anyone meeting that page for the first time still meets the same three questions.
The check is cheap and almost nobody runs it. When a number improves sharply without the underlying thing changing, ask who is on the other side of your boundary and whether they have noticed anything.
10 · The evidence
Maya: The settings page did not change. That is the sentence I should have noticed in week one.
11 · You make the call
The story does not tell you first.
A. The help article was badly written
B. The measurement boundary ran between support and sales
C. Six weeks is too short to see the effect
12 · What happened
The settings page gets rebuilt the following month. The article stays, and demo stalls fall to two a week.
Maya thinks: A ninety-one percent improvement that improved nothing.
13 · Complete
When a number improves sharply and the underlying thing did not change, look at who is on the other side of your boundary. Next: what happens when the feedback arrives three ships too late.