Engineering, production, governance, cost, communication · insight-0296

Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects

Situation
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
Action
Define the goal metric together with guardrails that catch likely gaming paths (repeat contacts, complaints, revenue per customer), review them together, and rotate or audit metrics when optimisation pressure is high.
Reason
Optimising a proxy beyond a point can make it ineffective or harmful through several mechanisms; guardrails reveal the damage early.
Boundary
Operational KPIs, model objectives, incentive schemes, experiment success metrics.

Symptom or risk

Once a metric becomes a target, teams and models optimise the metric rather than the goal, and side effects grow unnoticed.

When the metric is the goal itself (for example energy consumed), gaming risk is lower.

More metrics dilute focus; keep a small set with one primary.

Adding guardrails but never letting them block a launch.

Manheim and Garrabrant describe several distinct mechanisms by which optimising a metric makes it ineffective or harmful (Goodhart's law).
Categorizing Variants of Goodhart's Law
Kohavi, Tang and Xu discuss experiment metrics including an overall evaluation criterion and guardrail metrics.
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing