Out of Jurisdiction

Research has a way of getting people excited. A user says something with real conviction and the room nods along. A study comes back with a number so clean it feels like a win. That excitement is worth watching, because it's usually the moment a team starts claiming more than the research showed.

The most useful skill in research isn't running the study. It's knowing what the study can rule on — its jurisdiction. What a given method has the standing to conclude, and what sits just outside that, dressed up to look like it belongs. This goes wrong in two places: before you collect anything, and after. Same mistake both times, pointed in opposite directions.

Before you collect: what people can tell you

When you want to learn from users, the obvious move is to get a bunch of them in a room and listen. It's a reasonable instinct. You hear from a lot of people at once, everyone gets a say, and it's hard to argue with: they're telling you what they think, so who are you to wave it off? Brushing aside what users say to your face can feel like arrogance.

That's the honest case for it, and it's a good one. Two problems, though.

First, groups concentrate opinions instead of averaging them. In Solomon Asch's conformity experiments, people agreed with an answer they could plainly see was wrong about a third of the time, just because a few confident voices went first, and roughly three in four went along at least once. Add one person willing to disagree and the effect mostly falls apart.^[1] People aren't sheep. It's that what you hear from a group leans heavily on who talks first and who talks loudest, and piling more people in makes that worse, not better. A big open call doesn't hand you the room's opinion. It hands you a few people's opinion, turned up loud.

Second, even when someone is being perfectly clear, you're hearing what they say, not what they'll do.^[2] The gap between the two is one of the most reliable findings in our field. Someone will tell you, with total confidence and total sincerity, "I want this, I'd use it every day, you have to build it." Then in real use, they never touch it. They weren't lying. They believed it when they said it. People are just bad at predicting what they'll do from inside a hypothetical. This is old news, too. In 1934, Richard LaPiere drove around the country with a young Chinese couple and got turned away exactly once in more than 250 hotels and restaurants. He wrote to those same places afterward asking whether they'd serve Chinese guests, and over 90% said no.^[3] What people said and what they'd already done weren't just different. They were opposite.

And only the first of those is really a group problem. Sit one person down on their own and the say-do gap is just as wide. There's a related trap worth naming while we're here. People are the best source in the world on their own experience — nobody knows their problems, their frustrations, and their homemade workarounds better than they do. What they're not good at is speaking for everyone else. A user can tell you exactly where it hurts. They're much worse at knowing how common that pain is, and worse still at prescribing the fix. "You should build X" is a design decision in a user's mouth. Their problem is theirs to report. The solution, and the leap to "everyone needs this," isn't.

So here's where the line falls. Group or solo, people can tell you what they say, what they want in the moment, and what problems they keep hitting. That's worth a lot: it surfaces pain points, builds rapport, and points you at leads worth chasing. What it can't give you is what they'll actually do, how widespread any of it is, or the design itself. Treat what you get as hypotheses to check, not decisions to ship. Match the format to the goal. For listening and discovery, it's exactly right. For settling a design question, it's the wrong tool no matter how many people show up,^[4] because that calls for watching what people do, not collecting what they say.^[5]

After you collect: what a strong result buys

The second place we cross the line stings more, because it's about our own wins.

Say a study comes back strong, a near-perfect result on exactly what you set out to test. The temptation is to let it mean everything. And there's a fair case against over-hedging, one I take seriously. A clean result is rare and hard to get. Caveat every finding into oblivion and you start to sound like a coward, or like research that answers "it depends" to everything and never helps anyone decide. Stakeholders are right to be sick of that.

All fair. Which is exactly why the goal isn't to hedge the result to death. It's to be as clear about what it didn't show as about what it did.

Here's an example. A team runs a quick lab study on a new design that hides features a user doesn't have access to yet, so the screen only shows what's available to them. The question is whether people can still find their way around. They can: every participant gets it, a clean 100%. That part is solid. People learned the pattern, and learned it fast, with the first try the biggest jump. They applied it to screens they hadn't seen instead of memorizing one path. And when something they expected wasn't there, they read it right: "I don't have access to that yet," not "this is broken." The team watched it happen. You can build on that.

What it doesn't tell you is everything else that 100% makes you want to believe. It doesn't tell you support tickets will drop; nobody counted tickets, and you can't in a thirty-minute session anyway. It doesn't tell you how the design feels after six months, once the novelty is gone and habits have set in. It doesn't tell you whether hiding the features people don't have hurts upgrades, since people tend to buy less of what they can't see. And it doesn't tell you how anyone behaves with no moderator in the room, which is the only situation that matters in real life. The one people miss most: a clean result on the idea, that this kind of layout is learnable, says nothing about whether the exact screen you tested is any good. The concept held up. The specific labels, the placement, the layout are all still fair game. A win for the approach isn't a blank check for every decision inside it.

None of that makes the 100% any less real. It just stops people from spending it on things it never bought.

The point

So what's the actual point here, past "be humble"? Any piece of research is only worth what its method can support, and that limit never announces itself. Somebody has to say it out loud, on purpose.

And you're not doing this to be modest. You're doing it because a fuzzy finding is useless. If nobody knows how far a result reaches, nobody can act on it with any confidence. A result with a clear limit around it is one you can act on hard, because everyone can see where it stops. Naming that limit is what turns a finding into something you can build on without second-guessing it later.

It's also where your credibility comes from. Not the size of the number; a big number impresses people for a day. What earns you trust over the long run is being just as clear about what a result can't support, especially when everyone in the room, you included, wants it to support more. Your "yes, this one's solid" only carries weight because your "no, that part's out of jurisdiction" came first.

So make it a habit, every time, before anyone falls for the result: ask what it can actually conclude. Draw the line early, while it's easy, before the excitement makes it awkward to own up to what you left out. Then go all in on what's inside it. That part is real, and you earned it.