--- title: "Why 'We Talked to Customers' Is Not Research" description: "Most product teams confuse customer conversations with customer research. The distinction matters, and the cost of getting it wrong is measured in failed roadmaps." author: "Kody Everson" url: "https://theipp.org/insights/why-we-talked-to-customers-is-not-research" date: "2026-06-22T11:27:45.090Z" --- # Why 'We Talked to Customers' Is Not Research ## Summary Most product teams confuse customer conversations with customer research. The distinction matters, and the cost of getting it wrong is measured in failed roadmaps. ## Main content Walk into any product review and you will hear a sentence that has quietly become the most dangerous phrase in modern product development: _'We talked to customers.'_ It is offered as justification for a roadmap, a pivot, a feature cut, a pricing change. It closes debate. It signals diligence. And most of the time, it is not research. It is anecdotal feedback shared with a confident tone. This is the evidence standard problem. Product organisations have democratised customer contact, which is good, while simultaneously degrading what counts as evidence, which is not. The result is a generation of teams who believe they are evidence-led because they have a direct link to five users a month, and who cannot tell you why those five, what question was being tested, or what would have changed their minds. ## The mistake at the heart of modern discovery Continuous discovery was meant to fix a real problem: product teams making decisions in evidence vacuums, shipping based on stakeholder opinion and HiPPO instinct. The fix was simple enough, talk to customers every week. Build the muscle. Make contact a routine rather than ceremonial. The problem is that over time weekly customer conversations became a checkbox, then a shield. Teams started citing 'customer conversations' the way previous generations cited 'market research', as an unfalsifiable authority that ends arguments rather than informs them. Here is the uncomfortable truth. A conversation with a customer is an input. Research is a process that turns inputs into defensible claims. Confusing the two is like confusing ingredients with a meal. You can have all the ingredients in the world and still produce nothing edible. ## What actually distinguishes research from conversation The product profession needs to be honest about what separates evidence from anecdote. Four things, at minimum. ### 1\. A falsifiable question Real research starts with a question that could come back with an answer you did not want. 'Do enterprise admins find our permissions model confusing?' is a research question. 'Let's understand our users better' is a tourist itinerary. If your discovery cannot fail, it is not discovery. ### 2\. Sampling logic Who you talked to matters as much as what they said. Five conversations with whoever your team could grab this week is not a sample. It is convenience. Good research specifies the segment, the recruitment criteria, and what populations the findings can and cannot speak to. When a PM tells me 'users want X', my first question is always: which users, recruited how, against what criteria? ### 3\. Structured analysis Notes in a document are not analysis. Highlight reels in a Friday demo are not analysis. Analysis means coding, comparing, looking for disconfirming patterns, and being explicit about how you got from raw transcripts to the claim on the slide. Most product teams skip this entirely and jump straight from quotes to conclusions, which is how a single articulate customer ends up shaping a quarter's roadmap. ### 4\. A defined evidence threshold This is the one almost nobody does. Before you start, what would constitute enough evidence to act? What would constitute enough evidence to stop? Without a threshold defined in advance, every finding can be interpreted as either too thin to act on or strong enough to ship, depending on what the team already wanted to do. That is not evidence. That is rationalisation with extra steps. ## Why this matters more than it used to You could argue the bar has always been low and products still get built. True. But three things have changed. First, the cost of being wrong has gone up. AI features, platform shifts, and pricing model changes are large enough to be bet-the-roadmap decisions, often irreversible within a fiscal year. The casual evidence standards that worked when teams were shipping incremental B2B SaaS features do not survive with these decisions. Second, the volume of customer signal has exploded. Sales calls, support tickets, community Slack, in-app feedback, NPS verbatims, win/loss interviews, usage analytics. Teams are drowning in inputs and starving for evidence, because nobody has invested in the discipline that turns one into the other. More data, lower confidence. That is a systems failure. Third, the credibility of product as a function depends on it. When product leaders walk into board rooms and claim customer-led decision-making, then cannot defend the basis of a single major bet under serious questioning, the function loses authority. Finance does not get to say 'we talked to some accountants' and call it a forecast. Engineering does not get to say 'we looked at some code' and call it an architecture review. Product should not get the pass it currently gets. ## The false consensus trap There is a particular failure mode worth naming. When evidence standards are weak, leadership teams routinely reach false consensus. The CEO heard from three enterprise prospects at a conference. The Head of Sales has a story about a churned account. The PM ran four user interviews. Everybody has 'talked to customers' but, everybody has different customers, different questions, and different interpretations. I have watched roadmaps get approved on this basis and then unravel six months later when the underlying evidence was finally inspected or the wind blows a different direction. The post-mortems always conclude that the team should 'do more research'. They rarely conclude that the team should have higher standards for what counts as research in the first place. That is the diagnosis that would actually change behaviour. ## A tiered evidence standard The answer is not to demand academic rigour for every product decision. That would be both impossible and stupid. The answer is to make evidence standards proportional to the reversibility and cost of the decision being made. A tiered feature model, perhaps. - **Tier 1, low cost and reversible.** UI copy, minor flow tweaks, experiment variants. Anecdote and intuition are fine. Ship it, measure it, move on. - **Tier 2, moderate cost or partially reversible.** New features inside existing screens, onboarding redesigns, pricing page changes. Requires structured discovery: defined question, deliberate sample of 8-15, coded analysis, pre-committed success criteria. - **Tier 3, high cost or irreversible.** New product lines, platform bets, pricing model changes, market entry. Requires multi-method evidence: qualitative depth plus quantitative validation, disconfirming research, explicit assumptions log, named decision owner and a documented basis of decision. This is not bureaucracy. It is the basic hygiene that every other serious decision-making discipline already practices. The point is to match the rigour to the stakes, not to apply the same superficial standard to everything and call it discovery. ## What product leaders should actually do this week Three concrete moves. First, audit your last five significant product decisions and ask: what was the actual evidence basis? Not the story that was told. The evidence. Who was talked to, recruited how, against what question, analysed how, against what threshold. If you cannot reconstruct it, that is the finding. Second, separate the activity from the evidence in how your teams report. 'We ran 12 customer conversations this sprint' is an activity metric. 'We have moderate-confidence evidence that mid-market admins abandon setup at the permissions step, based on N=14 structured interviews and supporting funnel data' is a backed up evidence claim. Third, invest in research as a discipline owned by product, not a service procured from a research team when budget allows. Every PM should be able to design a discovery study, defend a sample, code a transcript, and articulate a confidence level. If they cannot, that is a capability gap your function needs to close, not a hiring requisition for someone else. ## The standard the profession deserves Product management spent the last decade fighting for the right to be evidence-led rather than opinion-led. We largely won that fight. The risk now is that we squander the win by accepting evidence standards so loose that 'evidence-led' means nothing more rigorous than 'I once spoke to a user about this'. The fix is not more customer conversations. The fix is treating evidence as a discipline, with standards, methods, and thresholds that a serious profession can defend. Until we do, 'we talked to customers' will keep doing the work that real research should be doing, and we will keep wondering why our confident roadmaps produce disappointing outcomes. Talking to customers is necessary. It has never been sufficient. The sooner the profession internalises the difference, the sooner we earn the authority we keep claiming we already have. ## Related pages - [Insights](https://theipp.org/insights.md) - [Product Profile](https://theipp.org/tools/product-profile.md) - [Standards](https://theipp.org/standards.md)