Statistics, experimentation, causal reasoning · insight-1030
Check that a proxy is a cause of the goal and not a consequence of it before making it a target
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry, telecom, banking
Summary: situation, action, reason, boundary
Situation
Hypothetical: call-centre staff are rewarded for short handling time, which correlates with satisfied customers.
Action
Draw the causal path from proxy to goal. If the proxy is an effect or only a correlate, do not target it; test intervening on the proxy in a small experiment and track the goal directly before scaling the incentive.
Reason
When the causal link between proxy and goal is indirect, intervening on the proxy need not move the goal, and optimising it can destroy the correlation that made it useful.
Boundary
KPI design, incentive schemes and model objectives with surrogate labels.
Symptom or risk
Satisfied, simple cases are quick, but pushing agents to be quick does not make customers satisfied and may cut off help that mattered.
Exceptions
Proxies that are on the causal path, such as seat availability for revenue.
Trade-offs
Direct goal metrics are noisier and slower.
Pitfalls
Justifying a target with a historical correlation alone.
Lessons and concepts
Claims and their sources
In causal Goodhart, when the causal path between the proxy and the goal is indirect, intervening can change the relationship so that the proxy no longer tracks the goal, and further intervention can be counterproductive.