When Two Metrics Disagree
A metric that moves before its outcome does is a hypothesis, not a verdict.
Short answer
When two metrics move in opposite or surprising directions, work through four steps before acting: name it, explain it, test it and close it. State exactly what the numbers say, list about three mechanisms including the innocent one, name the cheapest query that confirms or clears each, then sequence the checks and give branching recommendations.
Key takeaways
- Name it: state what the numbers are and are not saying before explaining anything.
- Explain it with about three distinct mechanisms, and include the innocent one.
- For each mechanism, name the cheapest query and what result would confirm or clear it.
- Run a magnitude check: the confirmed mechanism must be big enough to move the number.
- Close with one action set per verdict, each with a recovery signal to watch.
Common questions
What should I do when two metrics disagree?
Follow four steps. Name precisely what the data says, explain it with a few candidate mechanisms, test each with the cheapest decisive query, and close by sequencing checks and giving branching recommendations, each with a recovery signal.
What is the magnitude check in metric analysis?
A confirmed mechanism still has to be big enough to move the number you saw. Real congestion can be too small to shift a city average by two stars. Skipping this step means shipping a fix for a cause that explains only a fraction.
When should I use this approach?
Use it when an experiment ships, one metric jumps, another sags, and someone wants to roll back today. Correlation names a suspect, and clearing the alternatives names the cause. Watch especially when the metric that moved is the one the system was optimised on.
CheatsheetLayout, Section, NumberedItem, Callout, } from "@/components/CheatsheetLayout";
The genre
Two metrics move in surprising directions at once and you have to work out what is happening before you act. The trap is treating a shared timeline as a verdict, or treating a proxy gain as the real outcome. Measuring A, getting B. The proxy splits from the outcome it was meant to stand in for. Watch for it whenever the number that moved is the one the system was optimised on.
The four steps
Name it
State precisely what the numbers are and are not saying. Often the data already breaks the case. A metric moving the wrong way during its own supposed grace period is diverging, not lagging.
Explain it
Enumerate the distinct mechanisms that could produce the pattern, usually about three, and include the innocent one.
Test it
For each mechanism, name the cheapest query, and state both what result would confirm it and what would clear it. A signal that can only confirm and never clear is not a signal. It is a rationalisation wearing a metric's clothes.
Close it
Sequence checks cheapest and most decisive first. End on the real question, usually whether the system is rewarding the proxy or the outcome. Give branching recommendations: one action set per verdict, each with a recovery signal to watch.
The magnitude check
A confirmed mechanism still has to be big enough to move the number you saw. Real congestion can be too small to shift a city average by two stars. Skipping this step is how teams ship a fix for a cause that only explains a fraction of the movement.
Use it when
An experiment ships, one metric jumps, another sags, and someone in the room wants to roll back today. Correlation names a suspect. Clearing the alternatives names the cause. Source: Lewis Lin, M41H and M46H hidden signal cases (Jul 2026), building on Decode and Conquer. );