A/B Testing AI Features

Why experimentation on probabilistic systems is different — and the three methods (offline eval, shadow, online) that actually work.

CheatsheetLayout, Section, NumberedItem, Callout, CodeBlock, } from "@/components/CheatsheetLayout";

Why A/B testing AI is different

Traditional A/B testing assumes deterministic outputs. Same input → same output. You change a button color, you measure clicks. Clean, repeatable, statistical. AI systems are probabilistic. The same input can produce different outputs on consecutive calls. There's no "control" in the traditional sense because the system itself is non-deterministic. This creates three problems that don't exist in classical experimentation: You can't just run a feature flag. Exposing 50% of users to "the AI" vs. "no AI" tells you if the AI is better than nothing. It doesn't tell you if your changes to the AI are better. Output quality is subjective. There's no click event for "this answer was good." You have to define what good looks like before you can measure it. Model changes happen underneath you. If your AI provider updates their model mid-experiment, your results are contaminated. In classical A/B testing, you measure user behavior. In AI experimentation, you measure output quality first — then connect it to user behavior. Skip the first step and your experiment tells you nothing.

Three ways to test AI changes

Offline eval comparison (fastest)

Run Prompt A and Prompt B against a fixed dataset of 100–500 representative inputs. Score both using your eval suite. Compare scores. When to use: Prompt changes, model swaps, system prompt iterations. Gets you a directional answer in hours, not weeks. What you need: A representative golden dataset + an LLM-as-judge or code-based scorer you trust.

Offline eval flow:
1. Curate 200 real user inputs (varied, representative)
2. Run both versions against the same inputs
3. Score each output: pass/fail or 1-5 scale
4. If Version B wins by >5%, ship it
5. If delta is <5%, too close to call — run online test

Shadow mode testing

Run your new AI version in parallel with the current version. Both process every request. Only the current version's output reaches users. You compare outputs silently. When to use: High-stakes changes where you can't afford to expose users to a degraded experience — medical, financial, customer-facing support. What you need: Infrastructure to fork requests and store both outputs for comparison.

Online A/B test (last resort)

Expose a % of users to the new AI version. Measure acceptance rate, bypass rate, and satisfaction scores. Run until you hit statistical significance. When to use: When you need to validate that offline quality improvements actually translate to user behavior change. Warning: Online AI tests require large sample sizes because the effect sizes are often small. Expect weeks, not days. Most AI improvements don't need online A/B tests. If your offline eval shows a clear quality win, ship it. Save online tests for changes where the quality signal is ambiguous and user behavior is the only truth.

The eval-driven experiment setup

Before you can test anything, you need an eval suite that measures what matters. Building this is the real work.

Define failure modes before metrics

Don't start with "I want to improve quality." Start with "here are the 5 ways this AI fails today." Manual review of 100 production examples using open coding (free-form notes) followed by grouping into patterns is the fastest path. From Hamel Husain & Shreya Shankar: "The output of this process is not a metric. It's a taxonomy of failures. Your metrics come from that taxonomy."

Build two types of evaluators

Code-based evals: Objective, rule-based. Is the output valid JSON? Does it contain the required fields? Is the response under the latency SLO? These are fast, cheap, and catch regressions reliably. LLM-as-judge evals: Subjective quality. Is the tone appropriate? Does the answer address the question? These are slower and more expensive but catch nuanced failures code can't detect.

Validate your judge before trusting it

An LLM judge that always says "pass" on a 99%-passing system is 99% accurate and completely useless. Measure TPR and TNR separately:

True Positive Rate (TPR):
Of all examples that should PASS, what % does the judge correctly identify?
True Negative Rate (TNR):
Of all examples that should FAIL, what % does the judge correctly catch?
Target: TPR > 85% AND TNR > 80% before relying on the judge

Get there by having a human expert label 100–200 examples, then measuring your judge's TPR/TNR against human labels.

The metrics that matter in AI experiments

Acceptance rate

The % of AI suggestions users act on without modification. GitHub Copilot benchmarks at ~35%. Your target depends on your use case — don't chase 100%. From Ryan Salva at GitHub: "Services built using LLMs are probabilistic. Your job isn't to achieve 100% acceptance. It's to define what 'good enough' acceptance means for your segment."

Bypass rate

The % of times users ignore the AI and use the old method instead. Rising bypass rate is your earliest warning signal that the AI is degrading or losing trust.

Correction rate

The % of AI outputs users edit before using. High correction rate = AI is directionally right but not precise enough. Useful for measuring prompt iteration progress.

Eval score delta

The change in your LLM judge's score between Version A and Version B on your golden dataset. Your north star for offline experiments.

When NOT to A/B test AI features

The change is a clear best practice

If you're fixing a factual error, removing a harmful output, or addressing a clear UX bug — just ship it. Running an experiment to confirm that removing hallucinations is good is a waste of time.

You'd need 60,000+ users per variation

If your effect size is small and your traffic is limited, you'll be running the experiment for months. Use offline evals instead and accept directional signal over statistical certainty.

You're launching something entirely new

You need a control to A/B test. If you're launching a net-new AI feature, define independent success criteria (target acceptance rate, target bypass rate) instead of a comparison test. Offline eval → shadow test (if high stakes) → ship. Run an online A/B test only when the offline signal is ambiguous and user behavior is the decisive variable. Sources: Lenny's Newsletter — "Building Eval Systems That Actually Improve Your AI Product" (Hamel Husain & Shreya Shankar), "Beyond Vibe Checks" (Aman Khan), "Why Your AI Product Needs a Different Development Lifecycle" (Aishwarya Reganti & Kiriti Badam). GitHub Copilot acceptance rate data from Ryan Salva. );