Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Write down exclusions, covariates and the stopping rule before unblinding, because each free choice is a hidden test
After a flat result, an analyst tries excluding bot-like sessions, adding a covariate, trimming the top 1%, and switching to log revenue, and reports the variant with p = 0.03.
easyempirical finding - Do not read a market-level correlation as a statement about how individual customers behave
Across 60 markets, higher average ancillary spend goes with a higher business-traveller share, and a manager concludes that business travellers spend more on ancillaries.
easydocumented fact - Write the decision table before the result arrives, mapping each interval to an action
A test finishes with a lift of +0.4% and an interval from -0.3% to +1.1%, and the room argues about whether that is a win.
easypractitioner heuristic