How to Build AI Product Sense

A 15 minute weekly ritual that shows you where a model breaks before your users find out.

CheatsheetLayout, Section, NumberedItem, ExampleSection, Callout, } from "@/components/CheatsheetLayout";

What AI product sense is, and why a ritual

AI product sense is knowing what a model can do, where it fails, and how to build something people love inside those limits. Meta now interviews PM candidates on it: how you work with uncertainty, notice when the model is guessing, and still decide. Part 1 of this series (Tal Raviv and Aman Khan) builds the intuition by doing non technical work inside Cursor. Part 2, by Marily Nika, turns it into a practice for your own product. Her recurring pattern: a feature works in a controlled flow, then breaks in production on a few predictable failure modes, like a support agent that confidently answers "Is this good?" instead of asking what the user means. So once a week, run three rituals on whatever AI workflow you are building. Together they take under 15 minutes. Map the failure modes (rituals 1 to 3). Define the minimum viable quality (MVQ) and cost envelope. Design guardrails where behaviour breaks.

Ritual 1: ask for something obviously wrong (2 min)

Goal: see the model force structure onto chaos. Take messy, half formed data (Slack threads, meeting notes, Jira comments) and ask the model to extract "strategic decisions". Faced with mess, generative models confidently invent structure. Seven messages from four people: a possible Stripe failure for EU users, a guess that it is the webhook, a plea not to rename the onboarding modal again, an open dark mode question, "onboarding out by Thursday", a banner maybe still broken on mobile, and "I can fix the copy later". Asked for strategic product decisions, the model produced a clean roadmap, assigned the wrong owners, and turned offhand comments into commitments. Authoritative, structured, and wrong.

Re-run the same thread

Same messy context, a prompt like "draft our Q4 roadmap from this discussion". If it invents features nobody discussed, that is a failure mode.

Tell it what good looks like, run again

Add one line: include only items explicitly mentioned in the thread, and reply "Not enough information" where something is missing. The trustworthy answer admits there are no clear decisions, asks clarifying questions, surfaces themes, and assigns no owners unless stated.

Compare the two outputs side by side

Which constraint fixed the hallucination? What does the model need: explicit constraints, better context, tighter scope? Does the good version feel shippable or still brittle? What would the user experience in each version?

Capture the gap as a product requirement

A repeating failure mode points to a specific product gap and fix. Write it down.

Ritual 2: ask for something ambiguous (3 min)

Goal: find the model's semantic fragility, where it understands your words but misses your intent. When intent is unclear a probabilistic system fills the gap with its best guess, and that is where trust cracks. Upload a PRD and ask it to summarise for execs and list the top 5 risks and open questions. Does it over summarise, latch onto one irrelevant detail, ignore caveats, or assume the wrong audience? Other tells: you ask for UX problems and it proposes a new pricing model. Where the model gets confused is where the product should step in: ask the user to choose a goal ("Summarise for whom?"), give the model more context, or constrain the action so it cannot drift. You are not tricking the model, you are finding where communication breaks so design can prevent it.

Ritual 3: ask for something unexpectedly difficult (3 min)

Goal: find the first point of failure. Pick one task that feels simple to a human PM but stresses reasoning, context or judgement. You want to see where it breaks first, because that is where the product needs structure: guardrails, narrower inputs, or the task split into smaller steps. "Group these 40 bugs into themes and propose a roadmap." "Summarise this PRD and flag risks for leadership." What you write down is intended behaviour, not the final solution. After three rituals you hold the list of design work needed before users can trust the output.

Define the minimum viable quality (MVQ)

Performance almost always drops once a feature leaves the controlled environment, so set the bar early and check against it throughout. A strong MVQ names three thresholds plus a cost envelope. Lab demos hit over 90% accuracy, then fell apart in a real home with a barking dog and a dishwasher running. Acceptable bar: identifies the speaker x% of the time in typical home conditions, and recovers gracefully when unsure ("should I use your profile or continue as a guest?"). Delight bar: behavioural signals rather than a number. Users stop repeating themselves; "No, I meant..." corrections drop sharply. Tests: two people talking over a background video, the 6 p.m. kitchen with kids and dishwasher, and a mid command correction ("10 minutes... actually, make it 5"). Do-not-ship bar: misidentifies the speaker more than y% of the time in critical flows (purchases, messages, personalised actions), or forces repeated attempts just to be recognised. Rule of thumb: 8 or 9 attempts out of 10 working without a retry in realistic conditions feels magical. 1 in 5 needing a retry erodes trust fast. The x and y are blank on purpose. Thresholds depend on strategic context (the source lists five factors). Two it spells out: phase, since a closed beta tolerates rough edges that feel broken in a broad launch, and the stakes of the flow. Ballpark the cost to run at scale before you fall in love with the demo: cost per call, triggers per user per month, the worst case power user, whether caching or a smaller model helps, and whether the maths works at 10x usage. Per call: about $0.02 for a 30 minute transcript. Average user, 20 meetings a month: about $0.40 per month. Heavy user, 100 meetings a month: about $2.00 per month. With caching and a smaller model for low stakes meetings: roughly $0.25 to $0.30 per user per month on average. At $0.30 per user per month with retention impact, a no brainer. At $5 with unclear impact, a business problem.

Design guardrails where behaviour breaks

A guardrail decides what the product does when the model hits its limits. Often the fix is a product rule, not a different model. A startup feature summarised Slack threads into decisions and action items. It worked until it began assigning owners when nobody had agreed to anything, sometimes the wrong person. One line added to the system prompt: "Only assign an owner if someone explicitly volunteers or is directly asked and confirms"; otherwise surface themes and ask the user what to do next. That constraint removed the biggest trust issue almost immediately. Turn each intended behaviour from the rituals into an explicit rule: prompt, UX, and fallback. Then make the bars measurable with How to Define Evals for an AI Feature, and keep the failure patterns from AI Limitations Every PM Must Know Cold in mind while you test. Push the model into its failure modes yourself, every week, before your users do. The question shifts from "is this a good idea?" to "how will this behave in the real world?" Source: Marily Nika, "Building AI product sense, part 2", Lenny's Newsletter (Feb 2026). );