Lesson 14: Evals Before Features
Rubric first, then red teaming, then the human feedback loop.
if (!lesson) throw new Error("Lesson metadata missing");
Section 1: The Core Idea
Non determinism breaks the measurement habits that work in deterministic software. Same prompt, two users, two different answers. One is accurate, one is fabricated. A conversion funnel scores both as a success, because both were delivered and neither was reported. If your only instrument is a funnel, you cannot tell a working AI feature from a confident one. Pre-AI, measuring quality meant picking a metric and running an A/B test. That still matters. But AI output is non-deterministic.
Section 2: The rubric comes first
Before you automate anything, define good output in writing, and name the ways it fails. What does an acceptable answer contain, and for whom? What are the named failure modes: fabricated facts, wrong tone, missing context, unsafe advice? Which failures are tolerable and which are never acceptable? Hand test cases against that rubric yourself. If you cannot score twenty outputs by hand consistently, an automated scorer will not rescue you. Write rubrics defining good AI output and failure modes. Test cases manually to understand quality before automating.
Section 3: The loop
Once the rubric exists, run the loop. Red team the named failure modes. Deliberately try to produce each one. A failure mode you have never triggered on purpose is a failure mode you do not understand. Design the human feedback loop. Decide who reviews, on what sample, at what cadence, and what happens to the disagreements. Unresolved disagreement is a rubric problem, not a rater problem. > KEY POINT: Rubric, then red team, then review loop. Skip the rubric and the other two produce opinions instead of evidence.
Section 4: Planning has to change too
Strict sprint commitments assume the ground holds still. With AI, model updates break features that worked last week and costs move mid cycle. Use flexible milestones instead, and keep one standing question on the table: what changes if the model improves twenty percent next month? Sometimes the honest answer is that half your roadmap becomes unnecessary. Better to find that out in planning than in a retrospective. The PM who can only operate inside structured sprints will keep producing clean process artifacts while the product drifts underneath them.
Section 5: Translate the risk three ways
The same residual risk has to be expressed differently depending on the room. Finance: inference cost per outcome, and how it scales. Sales: latency and what the demo will feel like on a bad day. Legal: hallucination exposure and what the failure looks like in front of a customer. State quality probabilistically: how often it works, under which conditions, at what confidence. That sentence is the deliverable. Source: Lewis Lin, "The New AI PM Player Card" (Apr 2026). **Apply It:** Take one AI feature you have shipped or used this week. Write a five line rubric for it: what good output contains, and three named failure modes. Find three inputs that break it. For each break, write the one sentence you would say to legal. buildContent= );