Statistics, experimentation, causal reasoning · insight-1034
Write the decision table before the result arrives, mapping each interval to an action
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, Data analyst
- Industries
- cross-industry, aviation, saas
Summary: situation, action, reason, boundary
Situation
A test finishes with a lift of +0.4% and an interval from -0.3% to +1.1%, and the room argues about whether that is a win.
Action
Before launch, list outcomes as intervals (clearly positive, inconclusive, clearly negative, harmful on a guardrail) with an agreed action for each, the cost of extending the test, and the value of the information it would buy. Review after the result against that table.
Reason
A decision depends on payoffs and reversibility, not on significance alone; fixing the mapping in advance removes room for motivated reading.
Boundary
Launch decisions, pricing changes, model replacement.
Symptom or risk
The same result is interpreted after the fact to support each person's preference, and the threshold for shipping drifts to whatever the data say.
Exceptions
Exploratory work with no decision attached.
Trade-offs
Writing it forces uncomfortable agreement up front.
Pitfalls
Writing actions so vague that every outcome fits.
Example
Hypothetical: ship if the lower bound is above -0.2% on revenue and the cost saving is real; stop if the lower bound is below -1%; otherwise extend two weeks (which covers +0.4% with an interval of -0.3% to +1.1%).