Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Run A/A tests on a new experimentation pipeline before trusting its first A/B result
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
mediumpractitioner heuristic - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Treat a clean pre-trend plot as weak support for parallel trends, not proof
A difference-in-differences analysis of a new airport lounge on one hub shows no significant pre-period difference in trends against other hubs, and the report calls parallel trends "confirmed".
harddocumented fact - With staggered rollouts, do not rely on a single two-way fixed-effects DiD coefficient
A self-service kiosk was rolled out to airports at different months, and the team estimates one coefficient with airport and month fixed effects.
harddocumented fact - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Show the synthetic control's pre-period fit and a backdated placebo before showing the effect
A synthetic control built from other airports estimates that a new hub incentive raised passenger numbers, and the deck shows only the post-launch gap.
harddocumented fact - Curate a small, relevant donor pool for synthetic control instead of including every available unit
A synthetic control for one route uses all 200 other routes as donors and only 12 months of pre-period data.
harddocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Simulate the false-ship rate of a probability-to-beat-control stopping rule before using it daily
A team replaces p-values with "probability that B beats A" and ships as soon as it exceeds 95% on any daily check.
harddocumented fact - Check how well the pre-period predicts the metric before counting on CUPED to save traffic
A test plan halves the duration "because we use CUPED", but the metric is first-visit conversion or the population is mostly new customers.
mediumdocumented fact - Randomise fare tests by departure or route when treated and control passengers draw on the same seats
A fare experiment splits sessions between a discounted and the standard fare on the same departures.
harddocumented fact - Leave a washout gap between switchback periods when a pricing or ranking change lingers
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
harddocumented fact - Count markets, not passengers, when price or capacity is randomised at the route level
A pricing change can only be applied per route, 14 routes are available, and the analyst computes p-values on millions of passengers.
mediumpractitioner heuristic - Analyse only users who could see the change, then dilute the result before forecasting the company-wide effect
A seat-map change reaches 6% of sessions; the all-user test is flat, while a triggered analysis shows +3% on the page.
mediumdocumented fact - Analyse a campaign by assignment first, and read engaged-only numbers with their assumptions attached
An email offer is randomised; only 20% open it, and the analyst compares openers in the treatment arm with everyone in control.
mediumpractitioner heuristic - Agree the no-worse margin before a migration test and size it as an equivalence test
Moving the fare engine to a cheaper stack, the owner asks the test to show that revenue per search is "unchanged".
mediumdocumented fact - Keep a long-term holdback to measure what a year of shipped wins really added
A quarterly deck sums the lifts of 14 individually significant experiments and claims +9%, while the dashboard moved +2%.
mediumpractitioner heuristic - Validate a short-term proxy against the long-term outcome in past experiments before deciding with it
Rebooking within 30 days is used as a proxy for annual customer value and changes ship when the proxy rises.
harddocumented fact - Estimate who responds with sample-splitting methods and confirm the top segments in a fresh randomised holdout
After a loyalty-offer test, the team fits outcome on treatment times 60 covariates on the same data to find "who responds".
hardpractitioner heuristic - Evaluate an uplift-targeting rule on a randomised holdout, not by the outcome of the people it selected
A model ranks customers by predicted response to a discount, the top decile converts at 14%, and the slide says the rule "earns" 14%.
mediumpractitioner heuristic - Choose the multiplicity family and error rate by what a false positive costs, and say which you used
A route-level analysis tests 80 hypotheses; one analyst uses Bonferroni, another raw p-values, a third Benjamini-Hochberg, and the reports cannot be compared.
mediumdocumented fact - Estimate many route or segment effects with partial pooling instead of testing each one separately
A test is read out for 40 origin markets with a few thousand passengers each, and the analyst lists those with a raw p below 0.05, or those that survive a correction.
harddocumented fact - Write down exclusions, covariates and the stopping rule before unblinding, because each free choice is a hidden test
After a flat result, an analyst tries excluding bot-like sessions, adding a covariate, trimming the top 1%, and switching to log revenue, and reports the variant with p = 0.03.
easyempirical finding - Expect the best of many arms to disappoint on rollout and re-estimate the winner in a fresh sample
A pricing-page test runs 10 variants, and the top one shows +3.1% and is shipped on that number.
harddocumented fact - Test for sorting around the cutoff before trusting a regression discontinuity estimate
A status tier starts at a points threshold (hypothetical: 25,000), and members just above are compared with those just below to estimate the effect of lounge access.
harddocumented fact - Fit regression discontinuity locally and show how the estimate moves with the bandwidth
A global fourth-degree polynomial in loyalty points gives a significant jump at the cutoff, while the fitted curve swings wildly at the edges.
mediumdocumented fact - Describe a regression discontinuity effect as local to the cutoff and do not extrapolate it to the whole base
The estimate says a status upgrade raises rebooking by 4 points for members near the threshold, and the sponsor wants to upgrade all 200,000 members.
mediumpractitioner heuristic - Test whether a difference-in-differences result survives a change of scale before reporting it
A difference-in-differences on passenger counts says treated routes grew less than controls, while the same data in logs says the opposite.
harddocumented fact - Date the treatment from the announcement and check controls for spillover before running a difference-in-differences
A fare change on one route was announced three weeks before it took effect, and a competing route serves as the control.
mediumdocumented fact - Report the first-stage strength of an instrument and expect a weak one to pull estimates toward the biased answer
A weather-disruption indicator (hypothetical) is used to instrument the fare a rebooked passenger paid, and the first-stage F-statistic is about 3.
harddocumented fact - Inspect propensity-score overlap and weight extremes before reporting an inverse-probability-weighted effect
An observational analysis of upgrade offers uses inverse-probability weights, and a handful of customers carry weights above 50.
mediumdocumented fact - Quantify how strong an unmeasured confounder must be before dismissing or trusting an observational effect
Customers who use app check-in show a 1.5 times higher ancillary purchase rate, and the room splits between "it works" and "it is just engaged customers".
mediumdocumented fact - Do not read a market-level correlation as a statement about how individual customers behave
Across 60 markets, higher average ancillary spend goes with a higher business-traveller share, and a manager concludes that business travellers spend more on ancillaries.
easydocumented fact - Check residual autocorrelation in an interrupted time series and widen the intervals if it is there
A weekly interrupted time series after a schedule change gives a narrow, significant level shift from an ordinary least squares fit.
mediumdocumented fact - Expect a noisy predictor's coefficient to shrink toward zero and measure its reliability before interpreting it
A regression finds that self-reported trips per year barely predicts loyalty spend, and the team concludes that frequency does not matter.
mediumdocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Report the effective sample size after weighting, because large weight variation shrinks it
A satisfaction survey is weighted to the passenger mix by route and cabin, and the report keeps quoting the raw respondent count in its margin of error.
mediumpractitioner heuristic - Check that a proxy is a cause of the goal and not a consequence of it before making it a target
Hypothetical: call-centre staff are rewarded for short handling time, which correlates with satisfied customers.
mediumdocumented fact - Define outcome metrics on matured cohorts when cancellations, refunds or no-shows arrive late
A booking-flow test reports net bookings after two weeks, but cancellations and refunds keep arriving for up to a month.
mediumpractitioner heuristic - Audit what is logged for each arm when a variant changes the tracking, not only the product
A new checkout variant replaces a page, and its event tracking was rebuilt by a different team.
mediumdocumented fact - Plan power for a ratio metric from numerator-denominator covariance by customer, not the spread across rows
A test targets revenue per booked passenger, and the sample size was computed from the standard deviation of the ratio across bookings.
harddocumented fact - Write the decision table before the result arrives, mapping each interval to an action
A test finishes with a lift of +0.4% and an interval from -0.3% to +1.1%, and the room argues about whether that is a win.
easypractitioner heuristic - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact