Statistics, experimentation, causal reasoning · insight-1001
Simulate the false-ship rate of a probability-to-beat-control stopping rule before using it daily
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- e-commerce, saas, aviation
Summary: situation, action, reason, boundary
Situation
A team replaces p-values with "probability that B beats A" and ships as soon as it exceeds 95% on any daily check.
Action
Replay the exact rule on simulated A/A data with your real daily traffic and number of looks, read off the false-ship rate, then adjust the threshold, add a minimum sample and a practical-effect condition, or both.
Reason
Bayesian monitoring is valid only under proper stopping rules, and a ship rule has error rates of its own that grow with the number of looks.
Boundary
Dashboards that show Bayesian win probabilities and let people stop on them.
Symptom or risk
The posterior statement may be coherent, yet the rule as a whole ships neutral changes much more often than the 5% the threshold suggests.
Exceptions
Cheap, easily reversed changes where a higher false-ship rate is an accepted cost.
Trade-offs
A stricter threshold or minimum duration delays launches of real wins.
Pitfalls
Simulating one look and assuming the result carries over to 28 looks.
Example
Synthetic simulation (4% baseline, 2,000 users per arm per day, flat Beta priors, no true difference, 28 daily checks, 3,000 runs): a ship rule of P(B>A) > 0.95 fired in about 24% of runs, against about 5% for a single look at day 28.
Lessons and concepts
Claims and their sources
Bayesian testing with continuous monitoring is valid when proper stopping rules are used, and the authors identify common stopping rules that are improper and invalidate results.