Statistics, experimentation, causal reasoning · insight-0007
Check sample ratio mismatch before reading any experiment metric
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, Data analyst, ML engineer
- Industries
- e-commerce, saas, aviation
Summary: situation, action, reason, boundary
Situation
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
Action
Run a chi-square goodness-of-fit test of observed counts against the designed split as the first gate. If it fails, do not interpret results; trace assignment, redirects, bot filtering, logging loss and triggering conditions until the cause is found.
Reason
Randomization guarantees comparable groups only if all assigned units are counted. A mismatch is a symptom of selection, much like a fever is a symptom of many illnesses.
Boundary
Every randomized online experiment, including ramp-ups with unequal designed ratios.
Symptom or risk
A split that is off by more than chance means users were lost or added non-randomly, so every metric comparison may be biased in an unknown direction.
Exceptions
Designs where unequal arrival is expected and modelled, such as time-varying ramp schedules analysed per stage.
Trade-offs
Investigating delays the decision; shipping on a broken split risks reversing a good or bad result.
Pitfalls
Re-weighting the groups to the expected ratio, which hides the missing users instead of explaining them.
Example
For the counts above, chi-square is about 11.3 with one degree of freedom, p below 0.001.
Lessons and concepts
Claims and their sources
A sample ratio mismatch (observed ratio differs from the expected one, testable with a chi-square test) usually implies severe selection bias that can invalidate experiment results.