Statistics, experimentation, causal reasoning · insight-0014

Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you

Situation
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
Action
For each guardrail, write the largest acceptable degradation and check whether the test can detect it. Where it cannot, use a non-inferiority bound, a longer test, a larger ramp, or a variance-reduced proxy, and say so in the review.
Reason
Absence of a detected change only reassures if the test could have detected the change that matters.
Boundary
Latency, error rates, refunds, complaints, cancellations and other rare or noisy harm signals.

Symptom or risk

Guardrails with high variance or rare events have little power, so a meaningful harm passes unnoticed and the review gives false comfort.

Guardrails monitored continuously in production with fast rollback may tolerate a less powered pre-launch check.

Powering guardrails can dominate test duration; prioritize the ones with irreversible costs.

Reporting guardrails as green because their point estimates are near zero while the intervals include serious harm.

Assuming that an underpowered metric had no change is a documented interpretation pitfall in online experiments.
A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments