AI Analysis You Can Actually Trust

Four ways AI quietly breaks your customer research, and the prompt fix for each one.

CheatsheetLayout, Section, NumberedItem, ExampleSection, CodeBlock, Callout, } from "@/components/CheatsheetLayout";

Why AI gets customer data wrong

Put the same transcripts into two models and you get two narratives, two sets of "evidence" and two recommendations, delivered with equal confidence. In practice you only see one, so you build on it. Interviews are messy. A participant says something at minute 8 and reframes it at minute 35. LLMs impose structure too fast: clean themes, best fit quotes, tidy summary, done. Surveys are not as structured as they look. 200 answers to "Why did you cancel?" carry no context. Metadata columns sit next to customer responses, and Sullivan has seen AI treat an internal note, "flagged for follow-up", as something the customer said. Which model for what. Claude gives the most depth and breadth but does not filter, so proven patterns and half formed hypotheses arrive on equal footing. Gemini and NotebookLM give fewer, better grounded themes and can read non verbal behaviour in video. ChatGPT is best at framing findings for stakeholders and least reliable for verbatim quotes. Sullivan's pick for analysis is Claude.

Failure mode 1: invented evidence

What it looks like

Two kinds of hallucination. Fully fictional quotes, which still happen on all three models. And Frankenstein quotes stitched together from different parts of an interview, which is especially common in ChatGPT. Prompts like "max. 100 words", or asking for one punchy representative quote of 12 words or fewer per theme, almost guarantee mash-ups.

Why it happens

LLMs generate text that is statistically likely; they do not retrieve it like a search engine. "Verbatim" is ambiguous too: exact characters, filler words, where the quote starts and stops. The model fills those gaps silently, and a citation like [P03, 14:30] can be invented along with the quote.

The fix: quote rules, then verification

First define what a valid quote is to you. Paste this into the analysis prompt:

Then, after the analysis, ask the model to check each quote against the source and label it VERIFIED, PARAPHRASE or NOT FOUND, giving the actual wording for any paraphrase. When Sullivan ran this on a ChatGPT output, a majority of the quotes turned out to be paraphrases. It takes about five extra minutes.

Failure mode 2: false or generic insights

What it looks like

Themes like "price is a factor in decisions" or "people value reliability". True, useless, and they could describe any wearables study. A churn cluster saying 18% need "more actionable guidance" does not tell you whether to build clearer metrics or workout plans.

Why it happens

LLMs hunt for consensus, so obvious patterns rise and rare but important signals sink. Training priors push price to the top of every churn analysis, and your own wording primes them: mention "pricing issues" and watch how many responses get coded as pricing. Survey answers make this worse. "It's not for me" could mean too expensive, too data heavy, one more device to charge, or I need a screen. Four different product decisions in one row.

The fix: context loading that guides interpretation

Three lines of objectives are too little and four paragraphs of backstory are no better. Load at least: Project context: scope and stakes. "Exploring whether to add a screen" is a decision; "doing customer research" is not. Business goal: the question the evidence must answer, such as whether a feature wins back churned users or alienates loyal ones. Product context: domain knowledge. "I want to see my data" means something specific for a screenless wearable competing with Apple Watch. Participant overview: who is speaking. "I need real-time data" from a churned Garmin switcher is not the same as from a loyal user. For surveys, also spell out the data structure: which column is the customer's voice, which are internal tags to ignore, and coding such as 0 = churned. Thirty seconds here heads off the silliest mistakes.

Failure mode 3: signal that does not guide a decision

What it looks like

"22 respondents mentioned wanting a screen." "Sentiment toward screen feature is 72% positive." Feels like signal, but it cannot tell you how many of the 22 would actually be retained by a screen, how many really meant better app UX or GPS, and how many have billing problems a screen is irrelevant to.

Why it happens

The model counts mentions and treats them all equally. It cannot tell that someone checking their phone ten times per workout and someone who went back to Garmin for trail navigation need completely different solutions.

The fix: few-shot calibration

Do not describe your scale, show it: one real example per level, plus why it belongs there. From Sullivan's solution fit scale for the screen decision: 1, screen would retain: "I check my phone 10 times per workout just to see my HR". Specific friction, screen directly solves it. 2, cheaper fix would retain: the app was too clunky to check mid workout. Sounds screen related, but the phone was present, so an app UX fix solves it without hardware. Levels 3 to 5 were engagement fix, operational fix (billing, support) and unrelated competitive loss. With calibration and context loading in place, the tally gave a clear answer: 30% would be retained by a screen. Build the scale once, reuse it across analyses.

Failure mode 4: contradictory insights

What it looks like

The themes are clear, the quotes compelling, the table ready for the deck. Nobody checked whether the "strong advocate" also described behaviour that says otherwise. Sullivan suspects this is the most common failure mode: the other three produce output that looks good, and skipping verification is what lets them through.

Why it happens

Expert analysts do multiple passes instinctively. LLMs are trained to produce coherent, helpful answers, not to flag their own uncertainty. The first pass is always a hypothesis; without a second pass you are shipping a draft.

The fix: a verification pass on every analysis

Pushed to review its own labelling, any LLM will find errors: some big missteps, some small exaggerations lumped into a pattern that does not quite fit. A few minutes here is cheap against a hardware investment made on evidence with holes.

Try it on your last analysis

Take an interview summary or survey clustering you already ran with AI and apply the verification pass from failure mode 4. Most people find at least one quote that does not exist, one buried contradiction, or one "pattern" built on a single response. That is the system working. Invented evidence: quote selection rules plus a verification step False or generic insights: context loading with project, goal, product and participants Signal that does not guide decisions: few-shot calibration with real examples and reasons Contradictory insights: a final pass that catches what earlier passes missed For running the interviews themselves, see How to Do User Research for AI Features. Nothing in a confident AI output signals "check me again". Treat every first pass as a hypothesis and verify quotes, contradictions and confidence before anything reaches a deck. Sullivan's trade: fifteen minutes of verification now, or six months building the wrong thing. Source: Caitlin Sullivan, "How to do AI analysis you can actually trust", Lenny's Newsletter (Feb 2026). );