Maya finds that teams using Boards retain three times better, pushes Boards onto everyone, and learns from a holdout that the feature never caused the retention — the kind of team that retains anyway is the kind that finds Boards
The Feature That Caused Nothing — Volume I, Understanding Humans Before Products · Seeing Reality · Two paths · Core. Maya finds that teams using Boards retain three times better, pushes Boards onto everyone, and learns from a holdout that the feature never caused the retention — the kind of team that retains anyway is the kind that finds Boards
01 · Opening
Maya finds that teams using Boards retain three times better, pushes Boards onto everyone, and learns from a holdout that the feature never caused the retention — the kind of team that retains anyway is the kind that finds Boards
02 · The setup
The retention chart splits clean down the middle. Teams that use Boards stay. Teams that never open it leave.
Maya thinks: Three times the retention. That's the biggest number I've ever found.
03 · The setup
The split is real. Nobody has miscounted anything.
90-day retention by Boards usage — Uses Boards weekly: 71%; Never opened Boards: 24%; Accounts measured: 8,940; Randomly assigned: no
04 · The evidence
Maya: So we add Boards to onboarding and retention goes up.
Sam: I can sell that number tomorrow.
05 · The evidence
Dev: Or the teams that were always going to stay are the ones who go looking for Boards.
06 · The evidence
Nobody assigned these accounts to a group. The users sorted themselves — and whatever made a team organised enough to adopt Boards in week one is the same thing that makes them still be there in week twelve.
The number is accurate and the action it suggests is unsupported. That gap is the whole lesson: correct data plus a self-selected group tells you who these people are, not what the feature did to them.
07 · The evidence
Priya: Two ways forward. Ship it to everyone, or hold a slice back and find out.
08 · The evidence
Who put them in the group? — If the user did, you have a description of a type of person. If you did, at random, you have a result.
Name one thing that could cause both — Team size, budget, how organised they already are. If you can name it in ten seconds, the correlation is probably it.
What would the sceptical version cost to run? — A holdout is usually cheaper than the roadmap you were about to spend on the wrong thing.
09 · The evidence
They hold back a tenth of new accounts at random. No Boards in onboarding, no prompt, nothing.
Maya: Six weeks. Then we look once.
10 · The evidence
Boards in onboarding tripled adoption. It moved retention by less than a point.
Holdout, six weeks — Boards adoption — treated: 63%; Boards adoption — held back: 19%; 90-day retention — treated: 38.1%; 90-day retention — held back: 37.6%
11 · The evidence
Adoption moved. Retention did not. The feature was a symptom of a good account, never a cause of one.
Maya thinks: I nearly spent two quarters on this.
12 · You make the call
The story does not tell you first.
A. The onboarding change failed to drive adoption
B. Teams who adopt Boards were already the ones who would stay
C. The 8,940 accounts were too few to detect the effect
13 · What happened
Maya: The holdout cost us six weeks and saved us two quarters.
Priya: Keep it. Make it the habit.
14 · Complete
When users sort themselves into a group, that group describes them — it does not prove what you did to them. Next: when three angry emails should change the roadmap, and when they should not.