Statistics, experimentation, causal reasoning · insight-0017
Run A/A tests on a new experimentation pipeline before trusting its first A/B result
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, saas, cross-industry
Summary: situation, action, reason, boundary
Situation
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
Action
Run identical-treatment A/A tests, or repeatedly re-split historical data into fake arms, and check that about 5% of tests are significant at the 5% level, that p-values are roughly uniform and that split ratios match the design.
Reason
Under a true null, a correct pipeline must reproduce its nominal error rate; deviations reveal broken assumptions before they cost a decision.
Boundary
New platforms, new metrics, new randomization units and changes to the analysis code.
Symptom or risk
Bugs in hashing, logging or variance formulas produce false positives or biased splits that look like product effects.
Exceptions
Mature platforms with continuous A/A monitoring need them only when metrics or units change.
Trade-offs
A/A tests use traffic and time without product learning; offline re-splits are cheaper but miss live assignment bugs.
Pitfalls
Running one A/A test and declaring the system sound; a single pass says little about a 5% rate.