Statistics, experimentation, causal reasoning · insight-1008
Agree the no-worse margin before a migration test and size it as an equivalence test
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry, saas, aviation
Summary: situation, action, reason, boundary
Situation
Moving the fare engine to a cheaper stack, the owner asks the test to show that revenue per search is "unchanged".
Action
Fix the smallest loss the business would not accept as a margin, size the test so the interval can fit inside it, and test both one-sided hypotheses (or check the interval against plus and minus the margin). If the interval is wider, say which loss cannot be excluded.
Reason
Failing to reject zero difference does not show absence of an effect; equivalence tests reject effects large enough to matter.
Boundary
Migrations, refactors, cost-saving changes and guardrail claims of "no harm".
Symptom or risk
A non-significant difference is read as equivalence although the interval could not rule out a 3% loss.
Exceptions
Changes that need only a directional win.
Trade-offs
Tight margins need much larger samples than a superiority test.
Pitfalls
Choosing the margin after seeing the interval.
Lessons and concepts
Claims and their sources
Equivalence tests (TOST) test against bounds set by the smallest effect size of interest, and a non-significant result cannot show that an effect is absent.