Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Treat a clean pre-trend plot as weak support for parallel trends, not proof
A difference-in-differences analysis of a new airport lounge on one hub shows no significant pre-period difference in trends against other hubs, and the report calls parallel trends "confirmed".
harddocumented fact - With staggered rollouts, do not rely on a single two-way fixed-effects DiD coefficient
A self-service kiosk was rolled out to airports at different months, and the team estimates one coefficient with airport and month fixed effects.
harddocumented fact - Show the synthetic control's pre-period fit and a backdated placebo before showing the effect
A synthetic control built from other airports estimates that a new hub incentive raised passenger numbers, and the deck shows only the post-launch gap.
harddocumented fact - Curate a small, relevant donor pool for synthetic control instead of including every available unit
A synthetic control for one route uses all 200 other routes as donors and only 12 months of pre-period data.
harddocumented fact - Simulate the false-ship rate of a probability-to-beat-control stopping rule before using it daily
A team replaces p-values with "probability that B beats A" and ships as soon as it exceeds 95% on any daily check.
harddocumented fact - Randomise fare tests by departure or route when treated and control passengers draw on the same seats
A fare experiment splits sessions between a discounted and the standard fare on the same departures.
harddocumented fact - Leave a washout gap between switchback periods when a pricing or ranking change lingers
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
harddocumented fact - Validate a short-term proxy against the long-term outcome in past experiments before deciding with it
Rebooking within 30 days is used as a proxy for annual customer value and changes ship when the proxy rises.
harddocumented fact - Estimate who responds with sample-splitting methods and confirm the top segments in a fresh randomised holdout
After a loyalty-offer test, the team fits outcome on treatment times 60 covariates on the same data to find "who responds".
hardpractitioner heuristic - Estimate many route or segment effects with partial pooling instead of testing each one separately
A test is read out for 40 origin markets with a few thousand passengers each, and the analyst lists those with a raw p below 0.05, or those that survive a correction.
harddocumented fact - Expect the best of many arms to disappoint on rollout and re-estimate the winner in a fresh sample
A pricing-page test runs 10 variants, and the top one shows +3.1% and is shipped on that number.
harddocumented fact - Test for sorting around the cutoff before trusting a regression discontinuity estimate
A status tier starts at a points threshold (hypothetical: 25,000), and members just above are compared with those just below to estimate the effect of lounge access.
harddocumented fact - Test whether a difference-in-differences result survives a change of scale before reporting it
A difference-in-differences on passenger counts says treated routes grew less than controls, while the same data in logs says the opposite.
harddocumented fact - Report the first-stage strength of an instrument and expect a weak one to pull estimates toward the biased answer
A weather-disruption indicator (hypothetical) is used to instrument the fare a rebooked passenger paid, and the first-stage F-statistic is about 3.
harddocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Plan power for a ratio metric from numerator-denominator covariance by customer, not the spread across rows
A test targets revenue per booked passenger, and the sample size was computed from the standard deviation of the ratio across bookings.
harddocumented fact - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact