Statistics, experimentation, causal reasoning · insight-0032
Pair every target metric with a counter-metric that exposes gaming
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer, Applied AI
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
Action
For each target, name the most likely way to improve it without improving the outcome, and track a counter-metric that would reveal it (block time for on-time rate, bias and volume-weighted error for MAPE, repeat contacts for handling time). Review the pair together.
Reason
Optimization pressure on a proxy exploits the gap between proxy and goal; a counter-metric makes that gap visible.
Boundary
Incentive design, model objectives, OKRs and vendor SLAs.
Symptom or risk
People optimize the measure instead of the goal: schedules get padded, forecasts get biased toward easy items, calls get cut short.
Exceptions
Metrics that are the goal itself, such as cash collected, need fewer counter-metrics.
Trade-offs
More metrics dilute focus; keep one counter-metric per target.
Pitfalls
Choosing a counter-metric that the same team can also game.
Lessons and concepts
Claims and their sources
Optimizing a system against a proxy metric can fail through several distinct mechanisms once the metric is pushed hard (variants of Goodhart's law).