Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Do not choose intermittent-demand models by MAE because it rewards forecasting zero
On a mostly-zero series, an all-zero forecast wins the MAE comparison against every model.
harddocumented fact - Do not answer a price what-if by changing the price input of a demand forecast
A manager asks what bookings would be if fares were cut 5%, and the analyst re-scores the forecasting model with lower fares.
harddocumented fact - Model spill and unconstrained demand before estimating gains from added capacity
A planner asks for passenger gains if the aircraft on a route changes from 160 to 180 seats.
hardpractitioner heuristic - Evaluate an intervention against a counterfactual forecast with control series
A loyalty campaign launched on one market, and leadership wants to know whether bookings rose because of it.
harddocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact - Model a many-to-many relationship with a bridge table and state how weights prevent double counting
A booking can carry several promotion codes, and an analyst joins bookings to a promo column to report revenue by promotion.
harddocumented fact - Move fast-changing attributes into a mini-dimension instead of versioning the whole dimension
A customer dimension with millions of rows has a loyalty tier, risk band and credit-limit band that change monthly, and every change makes a type 2 copy of the whole row.
harddocumented fact - Store the zone name for a future local event, not the offset you know today
A schedule table stores departure times for next year as timestamps with a fixed offset computed at load time, and a government changes its daylight-saving rule.
harddocumented fact - Disambiguate repeated local times with an offset or fold before parsing wall-clock timestamps
A gate-system export has local timestamps without offsets, and on the night clocks go back the hour 01:00-02:00 appears twice.
harddocumented fact - Keep two clocks for corrected facts, when it was true and when you learned it
Finance restates last month's revenue after a refund batch is posted, and the dashboard owner overwrites the monthly figures; a model trained on last month's numbers can no longer be reproduced.
hardpractitioner heuristic - Set the late-data allowance from the measured arrival-delay distribution
A streaming job aggregates bookings by event time with an arbitrary 10-minute allowance, and the daily total disagrees with the nightly batch by a stable fraction of a percent.
hardpractitioner heuristic - Match near-duplicate entities with scored comparisons and a clerical-review band
Two passenger files should be merged, names and birth dates are keyed by hand, and exact-match deduplication leaves obvious duplicates while a loose rule merges different people.
harddocumented fact - Bracket a suspected MNAR column with a delta adjustment instead of arguing about MAR versus MNAR
A decision-critical column, such as stated income or a survey score, may be missing because of its own value, and the team debates the mechanism from missingness plots.
harddocumented fact - Judge an interaction on the decision's scale: no log-odds interaction can still mean unequal probability gaps
A logistic model has two drivers, such as booking window and market, and the team asks whether they interact using the p-value of the product term.
hardpractitioner heuristic - Replay intermittent-demand forecasts through a stock policy and report service and stock, not only error
Spare-parts demand is mostly zeros with occasional orders, and two methods tie on MASE.
hardpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic