The Single-Segment Testing Fallacy
Type: warning
Stage: Stage 4: Prototype Proof
Difficulty: intermediate
Testing five users only works if they are all the same kind of user. Mixing a buyer and an operator, an admin and an end user, produces contradictory data. When feedback is incoherent, the problem is almost always that you tested the wrong mix of people — not that the prototype is unfixable. The Rule of Five applies per segment, not per product.
Overview
Testing five users only works if they are all the same kind of user. The moment you mix two different user types into the same test — a buyer and an operator, an admin and an end user, a novice and an expert — you produce contradictory data that sends you in the wrong direction.
Why this happens
Founders recruit whoever they can get. They ask friends, reach out to their network, and take whoever says yes — without confirming that all five participants will use the product in fundamentally the same way. A usability session that includes both the person who configures a tool and the person who uses it daily will surface entirely different problems. The founder walks away with a list of contradictory issues and no clear priority.
This is not a sample-size problem. Adding more users does not fix it. Mixing segments produces noise regardless of how many participants you add.
The specific signs you have a segmentation problem
Watch for these after your test sessions:
— Feedback from different participants directly contradicts each other: one says the flow is too simple, another says it's too complex.
— You cannot identify a pattern across sessions — every user got confused at a different point.
— The same feature gets praised by some participants and criticized by others.
— You find yourself thinking "it depends on the user" when reviewing your notes.
When feedback is incoherent, the problem is almost always that you tested the wrong mix of people — not that the prototype is unfixable.
How to test whether you have a segmentation problem
After your sessions, write one sentence describing the person you tested. If you cannot write the same sentence for all five participants, you tested multiple segments.
Then ask: did the participants use the product to accomplish the same goal? If the answer is no — one was evaluating it for purchase, one was testing it for daily use, one was assessing it for their team — you have multiple segments and need to test them separately.
What counts as correct segmentation instead
Strong Stage 4 testing looks like:
— All five participants match a single, specific user profile: same role, same context, same goal.
— Feedback converges on the same two or three problems across sessions — meaning the issues are structural, not personal.
— You can prioritize your fixes with confidence because you know which problems affect the majority of your target user.
Identify your segments before you recruit. Test one segment completely before moving to the next. The Rule of Five applies per segment — not per product.