Maya surveys her active users to explain churn, ships the top request, and learns the people who could have answered had already left
The Users Who Aren't There — Volume I, Understanding Humans Before Products · Seeing Reality · Failure & recovery · Foundations. Maya surveys her active users to explain churn, ships the top request, and learns the people who could have answered had already left
01 · Opening
Maya surveys her active users to explain churn, ships the top request, and learns the people who could have answered had already left
02 · The setup
Maya has been a PM for three years. Churn is up four points this quarter, and nobody can say why.
03 · The setup
Priya: Churn is up four points. I need to know why before the board asks me.
04 · The evidence
Maya: I'll ask the users. A few hundred responses will tell us what's broken.
05 · The evidence
Four hundred and twelve responses. An average score of 4.6. One request louder than all the rest.
Q3 Satisfaction Survey (survey-results) — Responses: 412; Average score: 4.6 / 5; Top request: Bulk export
06 · The evidence
Sam: Four point six! They love us. Build the bulk export and churn takes care of itself.
07 · The evidence
So she built it. Six weeks of engineering, shipped on a Thursday, announced in the changelog.
08 · The evidence
Churn did not move. It got worse.
Priya: Bulk export shipped six weeks ago. Churn is up another point.
09 · The evidence
Four hundred people had told her what to build. She had built exactly that. It changed nothing.
10 · The evidence
Dev: Quick question. The survey — where did it actually run?
Maya: In the app. A banner after login.
11 · The evidence
The people who had left could not see a banner inside a product they had already stopped opening.
Dev: So everyone who answered was, by definition, still logging in.
12 · The evidence
Every study has a door. An in-app banner only reaches people who still open the app, so a survey about churn built that way excludes, by construction, every person it is about.
The 412 answers were not wrong. They answered a different question: what would make an already-satisfied user slightly more satisfied.
13 · The evidence
Who could have answered this? — Name the population the question is actually about — not the one you have access to.
Who actually answered? — Describe the sample honestly, including how they were reached. The method is part of the sample.
Who is missing, and would they disagree? — If the missing group would have answered differently, what you have is not evidence.
14 · You make the call
The story does not tell you first.
A. Users who found the survey annoying
B. Users who had already cancelled
C. Users on the free plan
15 · What happened
They talked about onboarding. All nine of them.
Maya: I emailed thirty cancelled accounts. Nine replied. Not one mentioned exports.
16 · Complete
Nine people changed her roadmap. Next: how many is enough, and when a sample stops being an anecdote.