AI Incident Response for PMs

When your AI feature breaks in production — the 4-step playbook, monitoring stack, and guardrail decisions every AI PM needs.

CheatsheetLayout, Section, NumberedItem, Callout, CodeBlock, } from "@/components/CheatsheetLayout";

AI failures are different from software failures

When a traditional feature breaks, you get an error log. The failure is discrete, traceable, and reproducible. You find the bug, fix it, deploy. AI failures are none of those things. They are: Silent. The system returns a response. No error thrown. Users see something wrong, not something broken. Non-deterministic. The failure may not reproduce on the next call. Asking "can you reproduce this?" is often the wrong question. Gradual. Quality doesn't crash to zero — it drifts. By the time someone notices, the problem has been live for days. Multi-layered. The failure could be in the prompt, the retrieval layer, the model, the context window, or your post-processing logic. Each layer needs separate diagnosis. "Is this a failure of capability (the AI can't do this) or a failure of configuration (the AI can do this but we haven't set it up correctly)?" These have completely different fixes.

The production monitoring stack

You can't respond to incidents you can't see. Before something breaks, you need three monitoring layers in place.

Real-time telemetry

Track the same signals you'd track for any production service: latency (p50, p95), error rates, token consumption, API timeouts. These catch infrastructure failures fast. From Intercom's engineering team: instrument your AI systems with Honeycomb or equivalent observability tooling. "You need real-time visibility into skill invocations to spot when AI-assisted features are degrading."

Async quality monitoring

Run your LLM-as-judge evaluator against a random sample of production outputs every hour or day. Track the score over time. A dropping score is your early warning signal — before users start complaining.

Quality monitoring cadence:
- P0 features: eval sample every hour
- P1 features: eval sample every 6 hours
- P2 features: eval sample daily
Alert threshold: >10% drop in eval score vs. 7-day baseline

Behavioral signals

Bypass rate and correction rate are your lagging indicators — but they catch failure modes that technical monitoring misses. A hallucination that sounds plausible will pass the eval judge but show up as elevated correction rate within 24 hours.

Guardrails: the PM decision framework

Guardrails are synchronous evaluators that block, redact, or regenerate responses before users see them. They're powerful but come with a real cost.

When to add a guardrail

Only when the cost of a false negative (bad output reaching users) is higher than the cost of a false positive (wrongly blocking a good output) + latency penalty. Add a guardrail: Legal liability, PII exposure, medical advice, financial decisions, content that violates ToS Don't add a guardrail: Stylistic quality issues, occasionally wrong answers in low-stakes contexts, slow or verbose responses

The three guardrail types

Block: Don't return a response. Show an error or fallback message. Use for safety-critical failures. Redact: Return the response with the problematic content removed. Use for PII, specific prohibited terms. Regenerate: Automatically retry with a modified prompt when quality falls below threshold. Use when latency budget allows. Every guardrail adds latency. Be ruthless about which ones are truly necessary. A guardrail that fires 0.01% of the time but adds 200ms to every request is a bad trade for most products.

The 4-step incident playbook

Isolate the layer

Before writing a single line of fix, diagnose which layer is failing. Each layer has a different owner and a different fix.

Multi-turn failure? → Test single-turn first
↳ Passes single-turn = conversational/context problem
↳ Fails single-turn = model or prompt problem
RAG system? → Evaluate retriever and generator separately
↳ Wrong document retrieved = retriever problem
↳ Right document, wrong answer = generator/prompt problem
Agent? → Map the transition failure matrix
↳ Where in the chain does it break most often?

Manual error analysis

Pull 20–50 failed examples from production. For each one, document: What the user said What the system did What should have happened Why it failed (prompt issue / model limitation / retrieval miss / missing context) Don't skip this step. Pattern recognition across failures is the fastest path to the right fix. A fix built on 3 examples often breaks 7 other things.

Decide: tweak vs. redesign

Tweak when: failures share a clear pattern, the fix is a prompt or parameter change, and the pattern covers <10% of inputs. Redesign when: failures span multiple components, fixing one failure introduces others, or the root cause is a capability the model fundamentally doesn't have. From Paul Adams (CPO at Intercom): "With AI, it is wholly unclear if something is possible to build. And when it is built, it is wholly unclear if it is any good, even if it appears to be good." Sometimes the honest answer is that you've discovered a capability boundary, not a bug.

Add to the CI safety net

Every production incident becomes a test case. Add the failure pattern to your golden dataset so it can never silently return.

After every incident:
1. Add 5–10 examples of the failure to your eval suite
2. Add a code-based or LLM-judge eval that catches it
3. Make it a CI gate — deploy blocked if this eval fails
4. Write a one-paragraph post-mortem: what failed, why, how you'd catch it earlier

The 4 hallucination types and how to catch them

Hallucinations are the most common AI incident. Each type has a different detection and fix.

Invented evidence

The AI generates plausible-sounding quotes, citations, or data points that don't exist. Fix: run a verification pass where the AI confirms each claim exists in source material.

False precision

The AI presents uncertain information with high confidence. Fix: require the AI to include hedging language and flag claims it can't verify.

Contradictory outputs

The AI produces internally inconsistent responses. Fix: add a final self-audit step asking the AI to check its own output for contradictions.

Plausible but wrong

The AI output looks correct but is factually wrong. Hardest to catch automatically — requires domain-expert human review on a sample basis. Catch rate improves with richer, more specific context in the system prompt. Instrument your AI like any production service. Track latency, error rates, and output quality. The difference: "error" in AI means "bad output," not "500 response." You need a quality metric to know when your error rate is climbing. Sources: Lenny's Newsletter — "Building Eval Systems That Actually Improve Your AI Product" (Hamel Husain & Shreya Shankar), "Why Your AI Product Needs a Different Development Lifecycle" (Aishwarya Reganti & Kiriti Badam), "How to Do AI Analysis You Can Actually Trust" (Caitlin Sullivan). Intercom engineering monitoring approach from Brian Scanlan. );