Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Run experiments in whole weekly cycles so the weekday mix matches across the decision
An experiment reaches its sample size on a Tuesday after starting on a Friday, and the team wants to close it.
easypractitioner heuristic - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Check sample ratio mismatch before reading any experiment metric
A 50/50 experiment shows 100,000 users in control and 98,500 in treatment, and the readout meeting is about to start.
easydocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Treat a p-value just under 0.05 as weak evidence that needs replication, not a verdict
A test ends at p = 0.04 on the primary metric and the readout says "statistically significant, ship it".
easydocumented fact - At very large sample sizes, judge the effect against the action threshold, not the p-value
A test with millions of sessions reports p < 0.0001 for a 0.02% conversion change, or an observational query over a year of bookings finds every coefficient significant.
easydocumented fact - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - State the base rate before quoting a detector's accuracy to stakeholders
A fraud or no-show alert is described as "95% accurate", and a manager plans staffing around its flags.
easyempirical finding - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Write down the grain of every table and assert it before the first aggregate
You start a KPI query on an unfamiliar table, or a table whose definition changed recently.
easydocumented fact - Compare row counts before and after every join to catch join explosions early
A query joins a flight-level table to a table with several rows per flight, and the resulting passenger total looks plausible but higher than finance reports.
easypractitioner heuristic - Aggregate each fact table to a shared grain before joining two facts together
You need bookings and ancillary revenue per flight, and both live in tables with many rows per flight.
mediumpractitioner heuristic - Put filters on the optional table in the ON clause, or your LEFT JOIN becomes an inner join
A query LEFT JOINs flights to promotions to keep all flights, then adds WHERE promo.channel = 'web'.
easydocumented fact - Decide whether NULL means zero before trusting AVG or COUNT(column)
An average fare or average delay is computed with AVG over a column that contains NULLs.
easydocumented fact - Expect SUM over an empty or all-NULL group to return NULL, and handle it at the edge
A dashboard shows blank revenue for a route-day with no sales, and a downstream ratio or growth calculation fails or disappears.
easydocumented fact - Use NOT EXISTS instead of NOT IN when the subquery column can contain NULL
A query finds customers who never booked with WHERE customerid NOT IN (SELECT customerid FROM bookings), and returns zero rows.
mediumdocumented fact - Specify ROWS and a unique ORDER BY in running totals, because the default frame groups ties
A running total of bookings uses SUM(...) OVER (ORDER BY bookingdate) and several rows share the same date.
mediumdocumented fact - Densify dates before a ROWS-based rolling window, because "7 rows" is not "7 days" when days are missing
A 7-day rolling average of passengers uses AVG(...) OVER (PARTITION BY route ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) on a table with gaps.
mediumdocumented fact - Partition lag features by entity and shift them by the forecast horizon, not by one row
A training table adds LAG(passengers) OVER (ORDER BY date) as a feature for a model that forecasts 14 days ahead.
mediumpractitioner heuristic - Report business metrics by event time and pipeline health by processing time
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
mediumdocumented fact - Mark recent periods as provisional until late-arriving data has had time to land
Yesterday's bookings look 15% low on the morning dashboard, and someone escalates a demand drop.
mediumdocumented fact - Make every load and backfill idempotent so a rerun cannot double-count
A failed nightly job is rerun manually, or a month of history is backfilled after a logic fix.
mediumdocumented fact - Deduplicate on a business key and keep the latest version by update time, not with DISTINCT
A raw table contains repeated rows for the same flight and date, some identical and some with a revised revenue value.
easypractitioner heuristic - Join facts to the dimension version that was valid at event time, not to current attributes
A route was reclassified from domestic to intl-short last year, and a revenue-by-market report joins all history to the current route table.
mediumdocumented fact - Keep facts whose dimension record has not arrived yet, using an explicit unknown member
Bookings for a newly opened route arrive before the route appears in the route dimension, and the reporting query uses an inner join.
mediumdocumented fact - Never sum across snapshots of a cumulative measure; pick one snapshot per entity
A booking-curve table stores cumulative bookings at several days before departure, and a query sums cumulativebookings per flight.
easypractitioner heuristic - Store instants in UTC and assign the business date with the local zone of where it happened
Bookings from several countries are aggregated to daily totals using the server's clock or a UTC date.
mediumdocumented fact - Expect 23-hour and 25-hour days in hourly data around daylight-saving changes
An hourly demand or call-volume series in local time shows a missing hour in spring and a doubled hour in autumn.
mediumdocumented fact - Normalize categorical keys and compare their cardinality with a reference list before grouping
A GROUP BY route returns more distinct routes than the network has, including near-identical codes.
easypractitioner heuristic - Check whether missingness depends on other columns before dropping incomplete rows
An analysis drops rows with missing fare or revenue because "only 3% are missing".
mediumdocumented fact - Distinguish "zero", "unknown" and "not applicable" instead of letting one NULL mean all three
A cancellations column is NULL for some flights, and analysts disagree on whether that means no cancellations, missing data or a flight that never operated.
easypractitioner heuristic - Classify an outlier as error, rare truth or regime change before deciding what to do with it
One flight shows ten times its usual passengers, or one day's sales are far above every previous day.
mediumdocumented fact - Encode impossible values as explicit integrity rules that quarantine rows and count them
EDA finds load factors above 1 and negative bookings in a raw table that feeds a forecast.
easypractitioner heuristic - Track distinct-value counts per column as an early warning for upstream changes
A model or dashboard depends on categorical columns fed by systems you do not own.
easypractitioner heuristic - Agree a data contract at the producer boundary before building on a shared table
Your forecast or KPI starts consuming a table owned by another team, with no written agreement about it.
mediumpractitioner heuristic - Check source freshness against an explicit SLA, more often than the SLA itself
A morning revenue dashboard occasionally shows yesterday's numbers unchanged because an upstream load failed silently.
easydocumented fact - Check row volume per entity and period to find gaps that schema tests cannot see
A table passes all type and null checks, yet a route's history has a hole of several days.
easypractitioner heuristic - Reconcile a new analytical table to an independent trusted total before anyone uses it
You have built a new revenue or passengers table for analysis or model training, and it will feed decisions.
easypractitioner heuristic - Compute rate KPIs as a ratio of sums unless you deliberately want the average of unit rates
Network load factor is reported as AVG(loadfactor) across flights, while finance computes passengers divided by seats.
easypractitioner heuristic - Cast to a decimal type before dividing integer counts in SQL
A conversion-rate query divides COUNT of bookings by COUNT of searches and returns 0 for every route.
easydocumented fact - Choose the chart from the comparison the reader must make, not from the data type
You need to show revenue share by fare level across three markets and the default tool suggests three pie charts.
easypractitioner heuristic - Start bar axes at zero, and switch to dots or lines when you need to show small differences
A bar chart of route load factors starts its axis at 75% so the differences look clearer.
easydocumented fact - Replace a spaghetti line chart with small multiples that share the same axes
A chart shows weekly bookings for twelve routes as twelve overlapping coloured lines.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Index series to a common base period instead of using a dual-axis chart
You want to compare growth of revenue (in millions) with passengers (in thousands) on one chart.
easypractitioner heuristic - Use a log scale for ratios and multiplicative growth, and label the original values
A chart compares route revenue that spans from thousands to millions, or shows growth multiples above and below 1.
easypractitioner heuristic - Flag capacity-constrained observations before treating sales as demand
A forecast or price analysis uses passengers or units sold as the target, and some flights sold out or some stores ran out of stock.
mediumempirical finding - Decompose a revenue change into volume, price and mix before explaining it
Quarterly revenue rose 8% and leadership asks why, while passengers, average fares and the route mix all changed.
mediumpractitioner heuristic - Use RobustScaler when a few extreme values distort mean-and-variance scaling, then decide about the extremes separately
A scaled feature has a handful of huge values, such as a 10x passenger outlier or a corporate group booking.
easydocumented fact - Do not give a linear model ordinal codes for unordered categories
A nominal feature such as market or payment method is label-encoded as 0, 1, 2 and fed to a linear, kernel or distance model.
easypractitioner heuristic - Avoid single mean imputation when the analysis is inferential rather than predictive
A stakeholder wants coefficients, confidence intervals or group comparisons from data with substantial missingness.
mediumdocumented fact - Replace sentinel values like -999 before they reach linear or distance models
A source system encodes missing or not-applicable as -999, 9999 or 0.
easypractitioner heuristic - Add explicit interaction terms to a linear model when the effect of one driver depends on another
Domain knowledge says an effect varies by context, for example price sensitivity that differs between short-haul and long-haul markets.
mediumpractitioner heuristic - Guard ratio features against zero and tiny denominators
You build ratios such as revenue per seat, cancellations per booking or spend per visit.
easypractitioner heuristic - Log-transform skewed inputs for linear and distance models, not for trees
A numeric feature spans orders of magnitude, such as corporate account spend or group size.
easypractitioner heuristic - Prefer splines to coarse bins when a linear model needs a nonlinear single-feature effect
You plan to cut days to departure or customer age into a few buckets so a linear model can bend.
mediumpractitioner heuristic - Compute entity history features as of the prediction timestamp with point-in-time joins
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
mediumdocumented fact - Remove features that are recorded because of the outcome, not before it
A candidate feature is filled in by a process that happens after or because of the target event, such as a cancellation reason, refund amount or collections flag.
easydocumented fact - Drop one dummy level for unpenalized linear regression, keep all levels for penalized models and trees
You one-hot encode categoricals for a linear model and must decide whether to set drop='first'.
mediumdocumented fact - Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - When reviewers can handle only K cases, rank and evaluate precision at K instead of choosing a probability cut-off
An operations team can investigate a fixed number of alerts per day regardless of how many the model flags.
mediumpractitioner heuristic - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Do not read lasso's choice among correlated features as evidence that the others do not matter
Lasso keeps one of several correlated features (competitor price, fare level, season) and zeroes the rest; someone proposes dropping them from the data.
mediumdocumented fact - Interpret predictions, not individual coefficients, when features are strongly collinear
Coefficients change sign or size when a correlated feature is added, removed or the data period shifts.
mediumdocumented fact - Treat a funnel-shaped residual plot as a modelling decision, not cosmetic noise
Residuals versus fitted values widen as predictions grow, common for revenue, demand and claim size.
mediumdocumented fact - Plot residuals in time order even for models that are not forecasts
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
mediumdocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Refuse to turn SHAP values into intervention advice without a causal design
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
mediumdocumented fact - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Plot ICE curves with the PDP so averages do not hide opposite effects
A partial dependence plot shows a flat or mild effect for a feature that domain experts expect to matter.
mediumdocumented fact - Test whether clusters survive resampling before naming them
A k-means run on customer or route features produces k tidy segments that the business wants to name and target.
mediumdocumented fact - Validate segments on outcomes they were not built from before calling them personas
A clustering of customers or routes is about to become a set of named personas with tailored strategies.
mediumpractitioner heuristic - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Widen or label scenario forecasts because they exclude uncertainty in the assumed drivers
Finance asks for next quarter's demand under three fuel-price scenarios, each with an 80% interval.
mediumdocumented fact - Report error by horizon and evaluate at the horizon the decision uses
A dashboard shows one pooled error for a 1-to-90-day forecast that feeds pricing decisions made 21 days before departure.
easydocumented fact - Guard MASE against zero denominators and state which naive scaling you used
MASE is computed across thousands of SKUs and some values come back infinite or enormous.
mediumdocumented fact - Compute WAPE at the level where actuals cannot sum to zero
WAPE is reported for every store-SKU-week cell and many cells are blank or infinite.
easypractitioner heuristic - Do not average MAPE across series of very different volumes
A portfolio report averages per-series MAPE, and a few slow movers make the whole forecast look broken.
easydocumented fact - Do not choose intermittent-demand models by MAE because it rewards forecasting zero
On a mostly-zero series, an all-zero forecast wins the MAE comparison against every model.
harddocumented fact - Separate stockout zeros from true zero demand before modelling a retail series
A store-SKU series looks intermittent, and stock records show the item was frequently unavailable.
mediumpractitioner heuristic - Reconcile hierarchical forecasts so route, region and network numbers agree
Route-level and network-level demand forecasts are produced separately, and the routes do not add up to the network total.
mediumdocumented fact - Match the forecasting level of a hierarchy to the noise and the decision
Store-SKU daily sales are too noisy to forecast well, but replenishment decisions are made per store and SKU.
mediumpractitioner heuristic - Check interval coverage by season and segment, not only overall
Overall coverage of a 90% demand interval is 89%, and the team declares it calibrated.
mediumpractitioner heuristic - Pick the forecast quantile from the ratio of under- and over-forecast costs
Operations asks for 'the forecast' to set meal loads per flight, where running out costs more than waste.
mediumdocumented fact - Handle weekly data's non-integer annual season with Fourier terms or STL
A weekly demand series is fitted with a 52-period seasonal model, and the holiday peaks drift by a week every few years.
mediumdocumented fact - Encode moving holidays as windows before and after, not as a single day flag
A Turkish domestic route forecast misses the travel surge around Eid holidays, which move about eleven days earlier each Gregorian year.
mediumdocumented fact - Model one-off events as interventions and keep them out of learned seasonality
A city hosted a single major sports final, and hotel and flight demand spiked for one week.
easydocumented fact - Use bookings already on hand as a known-at-origin input for departure-level forecasts
A revenue-management team forecasts final bookings for a departure 21 days out using only past departures.
mediumempirical finding - Shift before rolling so window features exclude the target period
A 7-day rolling mean of sales is computed and joined to the same row as the target.
easydocumented fact - Prefer simple models on very short series and say how little the backtest can tell you
A new route has 30 weeks of history, and someone proposes a seasonal ARIMA with many parameters.
easydocumented fact - Measure forecast value added at every step, including human overrides
Planners adjust the statistical forecast every month, and nobody knows whether the adjustments help.
easypractitioner heuristic - Track forecast bias separately from accuracy
A forecast with acceptable MAE consistently runs below actual demand on high-season weeks.
easypractitioner heuristic