Lesson 15: Trusting AI Analysis
Why an AI research summary is a hypothesis, not evidence, and the five minute check before you share it.
if (!lesson) throw new Error("Lesson metadata missing");
Section 1: The Core Idea
In Lesson 9 you pasted interview notes into a model and asked for patterns. This lesson is about what to do with the answer, because its biggest problem is invisible from inside it. Caitlin Sullivan, a user researcher, ran a simple experiment. Same customer transcripts, two models, same question: should a screenless fitness wearable add a screen? Back came two narratives, two sets of "evidence" and two recommendations, with equal confidence. One cherry-picked three enthusiastic quotes and said build it. The other refused to treat screen feedback as a literal feature request, split users by underlying need, and flagged pricing risk with timestamps you could check. Side by side you might spot the better one. In real life you only ever see one. The data is harder for a model than it looks. A 45 minute interview wanders; people say the important thing at minute 8 and reframe it at minute 35. A model resolves the mess by imposing structure fast. Surveys are worse. Two hundred answers to "Why did you cancel?" arrive with no context, next to internal metadata columns the model will happily read as the customer's voice. The rule: an LLM summary of research is a hypothesis about your data, not evidence from it.
Section 2: Failure mode 1: Invented evidence
What it looks like. Fully fictional quotes, which every major model still produces, and Frankenstein quotes stitched from different parts of an interview that capture the gist but are not the person's words. Ask for "a punchy quote of 12 words or fewer" and you will get mash-ups. Why. LLMs don't retrieve quotes like a search engine; they generate text that's statistically likely given the context. Generation and retrieval are fundamentally different. It predicts what a quote about phone-checking frustration should sound like; sometimes that matches the transcript, sometimes it does not. Citations too: a tag like [P03, 14:30] looks authoritative and can be made up along with the quote. What you do. First, define a valid quote for the model: start where the thought begins, keep hedges and emotional language, cite participant and timestamp, never combine statements from different parts of the interview. Second, ask it to check every quote against the source and label each VERIFIED, PARAPHRASE or NOT FOUND. On a ChatGPT output, Sullivan found most quotes were paraphrases.
Section 3: Failure modes 2 and 3: Generic themes and counts that decide nothing
What it looks like. Themes like "price is a factor" or "people value reliability": true, useless, and true of any product in your category. Or a count: "22 respondents mentioned wanting a screen". It feels like signal, but it cannot tell you how many of the 22 would actually stay for a screen, how many really meant better app UX, and how many have a billing problem a screen does nothing for. Why. Models hunt for consensus, and survey answers hide their meaning. "It's not for me" could mean too expensive, too much data for a casual athlete, one more device to charge, or I need a screen. One row, four product decisions. A model counting mentions treats "I check my phone 10 times per workout" and "I went back to Garmin" as the same signal, when one is solved by a screen and the other is not. What you do. Load context before you load data: the decision and its stakes, the business question the evidence must answer, enough product context to read "I want to see my data" correctly, and who the participants are. For a survey, name the column that holds the customer's voice. Then, if the output has to drive a decision, calibrate a scale by example: one real response per level and why it belongs there.
Section 4: Failure mode 4: Contradictory insights
What it looks like. Clear themes, compelling quotes, a table ready for the deck. Nobody asked whether the "strong advocate" also described behaviour pointing the other way. Sullivan thinks this is the most common failure mode, because the output looks finished. Why. An expert analyst makes several passes instinctively. A model makes one, and it is trained to be coherent, not to flag its own doubts. The first pass is always a hypothesis. Without another pass that specifically looks for errors, you're treating a draft as a final answer. What you do. Add a verification pass to every analysis: quote verification, a contradiction check per participant (stated preference versus described behaviour, confidence that softens later), and a confidence assessment that flags any finding resting on one or two voices. > KEY POINT: An AI summary of your research is a hypothesis about the data, not evidence from it. Nothing in the output will ever tell you to check it. Decide to check before you read the first line.
Section 5: The verification habit
Five minutes, before any research summary leaves your laptop. Context before data. Decision, business question, product context, participants. For surveys, the column map. Quote rules in the prompt. Where a quote starts, what it keeps, how it is cited, no combining. One example per level. If you are scoring responses, show each bucket with a real answer and the reason. Verification pass. Quotes, contradictions, confidence. Read the flags and revise. Spot check three quotes yourself. Open the transcript. If one is missing, check the rest. Verification is the difference between AI output you'll always second-guess and insights you can stand behind. Run it on something you already analysed and you will almost certainly find a quote that does not exist. That is the habit working. Source: Caitlin Sullivan, "How to do AI analysis you can actually trust", Lenny's Newsletter (Feb 2026). **Apply It:** Take any AI summary of customer feedback you already have. If you do not have one, paste 5 public reviews of a product you use into Claude and ask for the top 3 themes, each with a supporting quote. Then verify it: Check every quote against the original reviews. Label each one VERIFIED, PARAPHRASE or NOT FOUND. Pick one theme and recode it yourself. Read the reviews and decide which ones genuinely support it and which were lumped in. Write down how many quotes survived and whether the theme still holds. That gap is what you would have shipped without checking. buildContent= );