Error Discovery: Finding Hidden AI Failures

A 30 minute workflow to read your traces, annotate failures and pick which ones deserve an eval.

Short answer

Error discovery is the eval equivalent of product discovery: you read real user sessions to find which AI failures are worth measuring before you write any metrics. Collect traces, annotate about 10 by hand, then around 100 with an agent's help, cluster the notes into failure modes, and prioritise them by product impact. The loop takes about 30 minutes once you know the basics.

Key takeaways

  • Read traces before writing metrics, or you risk measuring the wrong thing.
  • Annotate 10 traces by hand first to guard against automation bias, then continue to about 100.
  • Write notes a colleague could act on, from the user's view, stopping at the first upstream error.
  • Prioritise failure modes by product impact, not just frequency, then turn each into a specific eval.
  • No traces yet? Log every AI session as one complete trace, or simulate queries using dimensions.

Common questions

What is error discovery in AI evals?

Error discovery is the eval equivalent of product discovery. It tells you which failures are worth measuring by having a human review real traces. The authors have run it with more than 50 companies and found major product flaws every time.

How many traces should I review before writing evals?

Review 10 traces by hand before the agent suggests anything, which guards against automation bias. After that, the agent proposes issues for you to accept or reject. Keep going until your own learning plateaus; the rule of thumb is 100 traces.

What is a trace in AI product evaluation?

A trace is one user session: the user input, the system prompt, every tool call and result, retrieved context, intermediate model calls and the final output. A reviewer should be able to reconstruct what happened and judge it. A local JSONL file is enough to begin.

CheatsheetLayout, Section, NumberedItem, ExampleSection, CodeBlock, Callout, } from "@/components/CheatsheetLayout";

Why you never skip error discovery

Most teams jump straight to writing metrics. Metrics feel concrete and automatable, while reading user sessions feels slow. But if you write metrics first you guess at what matters, and you end up measuring the wrong thing or the right thing badly. Error discovery is the eval equivalent of product discovery: it tells you which failures are worth measuring. The authors have run it with more than 50 companies, found major product flaws every time, and say the whole loop takes about 30 minutes once you know the basics. If you only have time for one part of the eval process, this is the part. Nurture Boss, an AI leasing assistant for property managers, had a prospect say the unit was out of budget. The assistant replied with a polite goodbye. Any agent would mark that a success. The product goal is to make sales, so the assistant should have offered cheaper units or sister properties. You only know to look for "objection handling" after you have seen the trace. Reviewing examples changing your definition of good is called criteria drift. In a study on 100 production traces from that assistant, agents missed issues needing product judgement (Markdown in SMS messages, missed human handoffs, objection handling), were good at obvious failures such as answers contradicted by tool output, and flagged good responses as failures. So the process is human first, agent second.

Step 1: start with your traces

A trace is one user session: the user input, the system prompt, every tool call and result, retrieved context, intermediate model calls, and the final output. A reviewer should be able to reconstruct what happened and judge it. A database, an eval vendor or a local folder all work; a local JSONL file is enough to begin.

No traces yet? Instrument the app

Ask your coding agent to log every AI session as one complete trace. If the app already sends traces to LangSmith, Arize, Phoenix or Langfuse, keep that and also write one JSON object per session to traces/traces.jsonl.

Not launched yet? Simulate user queries

Define a few dimensions you expect the product to fail on, then have the model turn each combination into a natural language query. Dimensions stop the output from being homogeneous. One model call per scenario gives more diversity than asking for everything at once.



A typical output: a renter with two dogs asking whether moving next month would work. Delete unrealistic examples, and if you spot gaps, add dimensions and regenerate. Real users beat synthetic data whenever you can get them.


Step 2: build the review app


The authors packaged the workflow as a free plugin. Install it and point your coding agent at your data.




evals-start routes you to the error-discovery skill, which reads a sample of your records to learn the schema, then builds and serves a small local review app shaped for your data. For the leasing assistant that is a message thread with tool calls alongside and a free text box for notes. For a writing assistant it is a text block with flags beside it. Show output the way the user sees it: an email looks like an email, a PDF renders as a PDF. Expose metadata you will filter on: for leasing, the channel (SMS, voice, web chat). Sample for diversity: the agent clusters traces and mixes cluster representatives with random picks. The big advantage of a coding agent is changing the interface on the fly. Missing field, a slice you want to filter to, a persona filter from your synthetic dimensions: just ask.


Annotate: 10 traces by hand, then 100 with help


The skill makes you review 10 traces before it suggests anything, a deliberate guard against automation bias. Your job is to leave a free text note on whatever bothers you: the assistant gave up instead of offering alternatives, the scheduling widget failed and it listed times instead, the tone is too formal for this persona.


Write it so a colleague could act on it


"Bad response" is useless. "Said the unit was available when the tool output showed it was leased" is actionable.


No root cause analysis


Describe what went wrong from the user's point of view. Do not try to diagnose retrieval problems yet.


Stop at the first upstream error


If a trace has several problems, note the first one. Upstream failures in agent trajectories matter more than downstream ones. Relax this once you are fast.


Failures only, to start


Annotating what makes a trace good helps the agent but can be skipped when short on time. After 10 annotations the agent proposes more issues for you to accept or reject. Do not blindly accept the early ones; use disagreements to correct its understanding, and expect a few rounds before it is useful. Keep going until your own learning plateaus. Rule of thumb: 100 traces, so you do not quit too early.


Step 3: turn failure modes into product priorities


With 100 or so annotated traces, the agent clusters notes into failure modes and counts them. For the leasing assistant the top two were "Failed to execute handoff" and "Interrupted customer". Frequency is not everything. Prioritisation is a product decision: losing a user's data outranks a slightly off tone. Objection handling appeared in about 25% of traces and directly hit revenue, so it became Nurture Boss's top priority. Each prioritised failure mode becomes a specific eval. A generic "politeness" metric would have scored the "You're welcome" goodbye highly and missed the lost tenant entirely. For turning a failure mode into a judge prompt, see Advanced Evals: Beyond Vibe Checks.


Where this sits in the eval loop


The earlier version of this process was mostly manual. Agents are now good at sampling data, building labelling apps and proposing annotations, so the human effort concentrates on judgement. The error-discovery skill itself is two Markdown files, kept minimal so it generalises across domains. The same plugin carries write-judge-prompt, validate-evaluator, evaluate-rag, eval-audit and build-review-interface for the later stages. Looking at your data is the highest ROI activity in building an AI product. No metric, dashboard or automated eval replaces the intuition you get from reading traces. The rest of the eval machinery exists to scale what you learn there. Source: Hamel Husain and Shreya Shankar, "Advanced evals: How to find (and fix) hidden AI failures in your product", Lenny's Newsletter (Sep 2026). );