Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Check how well the pre-period predicts the metric before counting on CUPED to save traffic
A test plan halves the duration "because we use CUPED", but the metric is first-visit conversion or the population is mostly new customers.
mediumdocumented fact - Count markets, not passengers, when price or capacity is randomised at the route level
A pricing change can only be applied per route, 14 routes are available, and the analyst computes p-values on millions of passengers.
mediumpractitioner heuristic - Analyse only users who could see the change, then dilute the result before forecasting the company-wide effect
A seat-map change reaches 6% of sessions; the all-user test is flat, while a triggered analysis shows +3% on the page.
mediumdocumented fact - Analyse a campaign by assignment first, and read engaged-only numbers with their assumptions attached
An email offer is randomised; only 20% open it, and the analyst compares openers in the treatment arm with everyone in control.
mediumpractitioner heuristic - Keep a long-term holdback to measure what a year of shipped wins really added
A quarterly deck sums the lifts of 14 individually significant experiments and claims +9%, while the dashboard moved +2%.
mediumpractitioner heuristic - Choose the multiplicity family and error rate by what a false positive costs, and say which you used
A route-level analysis tests 80 hypotheses; one analyst uses Bonferroni, another raw p-values, a third Benjamini-Hochberg, and the reports cannot be compared.
mediumdocumented fact - Write down exclusions, covariates and the stopping rule before unblinding, because each free choice is a hidden test
After a flat result, an analyst tries excluding bot-like sessions, adding a covariate, trimming the top 1%, and switching to log revenue, and reports the variant with p = 0.03.
easyempirical finding - Describe a regression discontinuity effect as local to the cutoff and do not extrapolate it to the whole base
The estimate says a status upgrade raises rebooking by 4 points for members near the threshold, and the sponsor wants to upgrade all 200,000 members.
mediumpractitioner heuristic - Date the treatment from the announcement and check controls for spillover before running a difference-in-differences
A fare change on one route was announced three weeks before it took effect, and a competing route serves as the control.
mediumdocumented fact - Inspect propensity-score overlap and weight extremes before reporting an inverse-probability-weighted effect
An observational analysis of upgrade offers uses inverse-probability weights, and a handful of customers carry weights above 50.
mediumdocumented fact - Quantify how strong an unmeasured confounder must be before dismissing or trusting an observational effect
Customers who use app check-in show a 1.5 times higher ancillary purchase rate, and the room splits between "it works" and "it is just engaged customers".
mediumdocumented fact - Do not read a market-level correlation as a statement about how individual customers behave
Across 60 markets, higher average ancillary spend goes with a higher business-traveller share, and a manager concludes that business travellers spend more on ancillaries.
easydocumented fact - Check residual autocorrelation in an interrupted time series and widen the intervals if it is there
A weekly interrupted time series after a schedule change gives a narrow, significant level shift from an ordinary least squares fit.
mediumdocumented fact - Expect a noisy predictor's coefficient to shrink toward zero and measure its reliability before interpreting it
A regression finds that self-reported trips per year barely predicts loyalty spend, and the team concludes that frequency does not matter.
mediumdocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Report the effective sample size after weighting, because large weight variation shrinks it
A satisfaction survey is weighted to the passenger mix by route and cabin, and the report keeps quoting the raw respondent count in its margin of error.
mediumpractitioner heuristic - Check that a proxy is a cause of the goal and not a consequence of it before making it a target
Hypothetical: call-centre staff are rewarded for short handling time, which correlates with satisfied customers.
mediumdocumented fact - Define outcome metrics on matured cohorts when cancellations, refunds or no-shows arrive late
A booking-flow test reports net bookings after two weeks, but cancellations and refunds keep arriving for up to a month.
mediumpractitioner heuristic - Audit what is logged for each arm when a variant changes the tracking, not only the product
A new checkout variant replaces a page, and its event tracking was rebuilt by a different team.
mediumdocumented fact - Write the decision table before the result arrives, mapping each interval to an action
A test finishes with a lift of +0.4% and an interval from -0.3% to +1.1%, and the room argues about whether that is a win.
easypractitioner heuristic - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact