The False Positive Trap: What an 89% Survey Score Actually Means
Type: case-study
Stage: Stage 1: Problem Proof
Difficulty: beginner
A founder surveyed 500 people, got 89% saying they 'definitely need it,' spent $100k building — and launched to 12 signups and $45 in revenue.
Overview
This case study is a warning. The founder did everything that looks like validation — large sample size, clear enthusiasm, strong stated intent — and it was all meaningless. Understanding why this happens is as important as knowing what to do instead.
The problem thesis
The idea: a new type of focus and productivity app for students. The problem — distraction, difficulty studying, inability to stay on task — is genuinely real and widespread. The market is large. The founder had good reasons to believe this was worth building.
The validation process
The founder surveyed 500 potential users. The survey asked about their productivity struggles and whether they'd use and pay for a solution.
The results looked remarkable: 89% said they 'definitely need' a better solution. 68% said they would pay $15/month for it. By any conventional reading of survey data, this was a green light.
The false signal
The founder interpreted this enthusiasm as product-market fit. They spent $100,000 and months of development time building the full application.
On launch day: 12 signups. 3 paid subscriptions. Total revenue: $45.
The gap between 68% stated willingness to pay and 0.6% actual payment rate is not a rounding error. It's a fundamental problem with the research method.
What went wrong
Every survey question asked about hypothetical future behavior: 'Would you use this?' 'Would you pay for this?'
People answering surveys about productivity apps are being polite. They want to be helpful. They also overestimate their own future discipline — the same cognitive bias that makes gym memberships profitable in January.
The survey measured what people wished were true about themselves. It didn't measure what they actually do. Past behavior and present payment are the only reliable measures of intent. Everything else is aspiration.
The lesson
Surveys measure interest. Only two things measure intent:
1. Past behavior — 'What did you do the last time this happened?' (the Mom Test)
2. Payment proof — clicking a real buy button, entering a card number, or handing over cash
Any validation method that lets a person be helpful without any cost to them will produce false positives. Make the test real. The only meaningful signal is behavior that costs something — time, effort, or money.