Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Size an A/B test from the smallest effect you would act on, not the effect you hope to see
A product owner asks how long to run a booking-page test and offers the uplift they expect, or no target at all.
mediumpractitioner heuristic - Do not stop a fixed-horizon A/B test the first time the dashboard turns significant
Stakeholders check the experiment dashboard daily and ask to ship as soon as the p-value drops below 0.05.
mediumdocumented fact - Name one primary metric and correct for multiplicity across the rest of the scorecard
An experiment scorecard shows 20 or more metrics and the readout highlights whichever ones moved.
mediumdocumented fact - Treat a winning segment found after a flat test as a hypothesis for the next test
The overall result is flat, so the analyst slices by country, device, fare class and tenure until one subgroup shows a significant uplift.
mediumdocumented fact - Plot the treatment effect by day of exposure before trusting a short-term uplift
A new seat-map design or prompt shows a strong uplift in a two-week test and the team wants to extrapolate it to the annual plan.
mediumdocumented fact - Report the numerator and denominator next to every ratio metric in an experiment
Click-through rate or revenue per search improves in treatment, and nobody has looked at clicks and searches separately.
mediumdocumented fact - Communicate a null result with the effects it rules out, not as "no impact"
A well-run test of a new upsell banner ends with p = 0.4 and a stakeholder asks whether the feature "does nothing".
mediumdocumented fact - Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Run A/A tests on a new experimentation pipeline before trusting its first A/B result
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
mediumpractitioner heuristic - Decide how to treat revenue outliers before the test starts, then apply it to both arms
Revenue per user in an experiment swings because a few corporate group bookings land in one arm.
mediumdocumented fact - When the total and every segment move in opposite directions, decompose rate and mix before concluding
Network load factor rose year on year while load factor fell on every individual route.
mediumpractitioner heuristic - Ask what removed units from the data before analysing the survivors
An analysis of "routes we still operate" or "customers active today" is used to judge which strategies work.
mediumpractitioner heuristic - Never read price elasticity off observational fares and bookings when prices respond to demand
A regression of bookings on average fare across flights shows a positive slope, and someone proposes raising fares to sell more seats.
mediumpractitioner heuristic - Leave mediators out of the model when you want the total effect of a decision
To estimate the effect of a loyalty email on spend, the analyst adds "sessions after the email" as a control variable.
mediumpractitioner heuristic - Model seasonality explicitly in an interrupted time series or the season will pose as the effect
A new ancillary bundle launched in May, and a before/after regression on monthly revenue per passenger shows a jump.
mediumdocumented fact - Look for other events at the intervention date before attributing a break to your change
Fares were changed on a route group in March, and an interrupted time series shows a sharp drop in load factor in the same month.
mediumdocumented fact - Read "X Granger-causes Y" as "X helps forecast Y", never as "changing X changes Y"
A report claims competitor fares Granger-cause your bookings and proposes reacting to every competitor change.
mediumdocumented fact - Expect the worst performers to improve without treatment and keep an untreated comparison
Operations selects the ten routes with the worst on-time performance last quarter for a new turnaround process, and reports their improvement next quarter.
mediumdocumented fact - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - Do not rank small groups by raw rates; show intervals or shrink toward the overall rate
A league table ranks sales agents, routes or stores by conversion rate, and the top and bottom places go to units with very few observations.
mediumpractitioner heuristic - Aggregate each fact table to a shared grain before joining two facts together
You need bookings and ancillary revenue per flight, and both live in tables with many rows per flight.
mediumpractitioner heuristic - Use NOT EXISTS instead of NOT IN when the subquery column can contain NULL
A query finds customers who never booked with WHERE customerid NOT IN (SELECT customerid FROM bookings), and returns zero rows.
mediumdocumented fact - Specify ROWS and a unique ORDER BY in running totals, because the default frame groups ties
A running total of bookings uses SUM(...) OVER (ORDER BY bookingdate) and several rows share the same date.
mediumdocumented fact - Densify dates before a ROWS-based rolling window, because "7 rows" is not "7 days" when days are missing
A 7-day rolling average of passengers uses AVG(...) OVER (PARTITION BY route ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) on a table with gaps.
mediumdocumented fact - Partition lag features by entity and shift them by the forecast horizon, not by one row
A training table adds LAG(passengers) OVER (ORDER BY date) as a feature for a model that forecasts 14 days ahead.
mediumpractitioner heuristic - Report business metrics by event time and pipeline health by processing time
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
mediumdocumented fact - Join facts to the dimension version that was valid at event time, not to current attributes
A route was reclassified from domestic to intl-short last year, and a revenue-by-market report joins all history to the current route table.
mediumdocumented fact - Build training rows with as-of joins so each feature uses only values known at the cutoff
A training table joins each departure to the latest customer and route features in the warehouse.
mediumdocumented fact - Store instants in UTC and assign the business date with the local zone of where it happened
Bookings from several countries are aggregated to daily totals using the server's clock or a UTC date.
mediumdocumented fact - Expect 23-hour and 25-hour days in hourly data around daylight-saving changes
An hourly demand or call-volume series in local time shows a missing hour in spring and a doubled hour in autumn.
mediumdocumented fact - Check whether missingness depends on other columns before dropping incomplete rows
An analysis drops rows with missing fare or revenue because "only 3% are missing".
mediumdocumented fact - Classify an outlier as error, rare truth or regime change before deciding what to do with it
One flight shows ten times its usual passengers, or one day's sales are far above every previous day.
mediumdocumented fact - Agree a data contract at the producer boundary before building on a shared table
Your forecast or KPI starts consuming a table owned by another team, with no written agreement about it.
mediumpractitioner heuristic - Flag capacity-constrained observations before treating sales as demand
A forecast or price analysis uses passengers or units sold as the target, and some flights sold out or some stores ran out of stock.
mediumempirical finding - Decompose a revenue change into volume, price and mix before explaining it
Quarterly revenue rose 8% and leadership asks why, while passengers, average fares and the route mix all changed.
mediumpractitioner heuristic - Standardize predictors before ridge, lasso or elastic net so the penalty is unit-free
You fit a penalized linear or logistic model where features are measured in different units.
mediumdocumented fact - Make target encoding fold-safe with cross fitting; fit().transform() on training rows leaks
You replace a categorical feature such as route or sales channel with the mean target per category.
mediumdocumented fact - Shrink target encodings of rare categories toward the global mean
Many categories have only a handful of rows, for example new routes or long-tail products.
mediumdocumented fact - Consider CatBoost's ordered target statistics when categorical encoding is the main leakage risk
A tabular problem is dominated by high-cardinality categoricals and your hand-built target encoding keeps leaking or overfitting.
mediumdocumented fact - Decide in advance how a model treats categories that first appear after deployment
A categorical feature grows over time: new routes, new products, new store codes.
mediumdocumented fact - Regularize LightGBM's native categorical splits when categories are many and small
You pass a high-cardinality integer-coded categorical to LightGBM as categoricalfeature.
mediumdocumented fact - Add a missing indicator when the fact of missingness may carry signal
A feature is missing more often for one segment or outcome, such as revenue missing more on intl-long flights in the synthetic raw table.
mediumdocumented fact - Avoid single mean imputation when the analysis is inferential rather than predictive
A stakeholder wants coefficients, confidence intervals or group comparisons from data with substantial missingness.
mediumdocumented fact - Monitor missing-rate shifts because a changed form or feed silently rewrites learned missingness
A model uses missing indicators or native NaN routing, and an upstream form, API or ETL job changes.
mediumpractitioner heuristic - Add explicit interaction terms to a linear model when the effect of one driver depends on another
Domain knowledge says an effect varies by context, for example price sensitivity that differs between short-haul and long-haul markets.
mediumpractitioner heuristic - Cap polynomial expansion degree and regularize, because generated features grow combinatorially
You apply PolynomialFeatures of degree 3 or more to dozens of inputs to capture nonlinearity.
mediumpractitioner heuristic - When you log-transform a skewed target, correct the back-transformed predictions toward the mean
You model log(revenue) or log(demand) and exponentiate predictions to report totals.
mediumdocumented fact - Prefer splines to coarse bins when a linear model needs a nonlinear single-feature effect
You plan to cut days to departure or customer age into a few buckets so a linear model can bend.
mediumpractitioner heuristic - Express time features relative to the decision moment, not as absolute dates
A model uses raw dates or timestamps as features, for example booking date or departure date as ordinal numbers.
mediumpractitioner heuristic - Compute entity history features as of the prediction timestamp with point-in-time joins
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
mediumdocumented fact - Run feature selection inside each training fold, never on the full dataset first
You screen hundreds of candidate features by correlation with the target or by model importance, then cross-validate the shortlist.
mediumdocumented fact - Normalize count and frequency encodings by window length so they survive changes in data volume
You encode a category by how often it appears, or an entity by its raw event count.
mediumpractitioner heuristic - Ship the fitted preprocessing with the model so training and serving transform rows identically
Preprocessing lives in a notebook while serving engineers reimplement it in another service or language.
mediumdocumented fact - Drop one dummy level for unpenalized linear regression, keep all levels for penalized models and trees
You one-hot encode categoricals for a linear model and must decide whether to set drop='first'.
mediumdocumented fact - Do not let a high ROC-AUC on rare events hide poor precision at the operating point
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
mediumempirical finding - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Treat a negative test R² as a signal of shift or broken features, not as a rounding curiosity
A regression model that scored well in validation shows negative R² on a new test period or segment.
mediumdocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Judge probability quality with log loss or Brier score, and calibration separately
Predicted probabilities will feed expected-value calculations, pricing or risk limits, and you only report ROC-AUC.
mediumdocumented fact - Check a reliability diagram before multiplying predicted probabilities by money
A boosted tree, random forest or SVM score is about to be used as a probability in an expected-cost or expected-revenue formula.
mediumempirical finding - Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
You choose a calibration method for a classifier with a modest validation set.
mediumdocumented fact - Recalibrate after resampling or class weighting, because both inflate predicted risk
You trained with SMOTE, undersampling, classweight='balanced', isunbalance or scaleposweight, and now use predictproba.
mediumempirical finding - Try the untouched data plus a cost-based threshold before class weights or resampling
Positives are rare and the instinct is to rebalance the training data.
mediumdocumented fact - Treat SMOTE as an experiment with known caveats, not a default fix for imbalance
Someone proposes SMOTE because the positive class is 3% of rows.
mediumempirical finding - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - Tune the threshold on cross-validated or fresh validation predictions, never on training predictions
You search thresholds to maximize a business metric such as profit or F-beta.
mediumdocumented fact - When reviewers can handle only K cases, rank and evaluate precision at K instead of choosing a probability cut-off
An operations team can investigate a fixed number of alerts per day regardless of how many the model flags.
mediumpractitioner heuristic - Report a tuned model's performance from nested cross-validation, not from the search's best score
You quote GridSearchCV.bestscore (or the best of many experiments) as the expected production performance.
mediumdocumented fact - Count every look at the test set as a selection step and refresh the holdout when looks pile up
A team checks the test score after each feature tweak, threshold change or model swap over several weeks.
mediumdocumented fact - Compare candidate models on identical folds and read per-fold differences before declaring a winner
Two models differ by a few thousandths in mean cross-validated score.
mediumpractitioner heuristic - Split by entity with grouped cross-validation when the same customer, aircraft or patient appears many times
Rows are repeated measurements of the same entity, and the model will score new entities in production.
mediumdocumented fact - Validate time-ordered tabular models on later periods, even when the task is not a forecast
A churn, credit or no-show model is trained on rows spanning two years and validated with shuffled K-fold.
mediumdocumented fact - Know that HistGradientBoosting switches early stopping on automatically for larger datasets
A HistGradientBoosting model behaves differently on a full dataset than on a sample, or training sizes vary across experiments.
mediumdocumented fact - Do not read lasso's choice among correlated features as evidence that the others do not matter
Lasso keeps one of several correlated features (competitor price, fare level, season) and zeroes the rest; someone proposes dropping them from the data.
mediumdocumented fact - Interpret predictions, not individual coefficients, when features are strongly collinear
Coefficients change sign or size when a correlated feature is added, removed or the data period shifts.
mediumdocumented fact - Treat a funnel-shaped residual plot as a modelling decision, not cosmetic noise
Residuals versus fitted values widen as predictions grow, common for revenue, demand and claim size.
mediumdocumented fact - Plot residuals in time order even for models that are not forecasts
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
mediumdocumented fact - Compare candidate and incumbent per slice before shipping an aggregate improvement
A new model beats the production model on the overall metric and a release is planned.
mediumpractitioner heuristic - Refuse to turn SHAP values into intervention advice without a causal design
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
mediumdocumented fact - Group correlated features before reading SHAP rankings
Several features measure the same thing (fare level, average fare, revenue per passenger) and SHAP ranks them separately.
mediumpractitioner heuristic - State the output scale of SHAP values for boosted classifiers before presenting them
You show SHAP values from a gradient-boosted classifier as percentage-point changes in probability.
mediumdocumented fact - Cluster correlated features before trusting permutation importance
Permutation importance on a held-out set shows every feature in a correlated group as nearly useless.
mediumdocumented fact - Prefer held-out permutation importance to impurity importance for tree models
You report the featureimportances attribute of a random forest or gradient-boosting model.
mediumdocumented fact - Plot ICE curves with the PDP so averages do not hide opposite effects
A partial dependence plot shows a flat or mild effect for a feature that domain experts expect to matter.
mediumdocumented fact - Encode known directional relationships with monotonic constraints in gradient boosting
Domain logic requires a direction (higher debt never lowers default risk, more days to departure never lowers remaining bookings), yet partial dependence wiggles against it.
mediumdocumented fact - Test whether clusters survive resampling before naming them
A k-means run on customer or route features produces k tidy segments that the business wants to name and target.
mediumdocumented fact - Do not use the silhouette score to choose between centroid and density-based clustering
You compare k-means and DBSCAN results using the silhouette coefficient and pick the higher score.
mediumdocumented fact - Validate segments on outcomes they were not built from before calling them personas
A clustering of customers or routes is about to become a set of named personas with tailored strategies.
mediumpractitioner heuristic - Do not assume the top principal components carry the predictive signal
You reduce features with PCA before a supervised model and keep components explaining 95% of variance.
mediumdocumented fact - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Set the contamination parameter from review capacity and a labelled sample, not as a guess of the anomaly rate
You configure IsolationForest or Local Outlier Factor and must choose contamination.
mediumdocumented fact - Drop or forecast any regressor that is not known at the forecast origin for the full horizon
A demand model uses yesterday's competitor fare as a feature, and the business needs forecasts 1 to 28 days ahead.
mediumdocumented fact - Report ex-ante accuracy, not ex-post, when a model uses future-unknown regressors
A team reports that adding temperature cut daily electricity-load forecast error by a third.
mediumdocumented fact - Widen or label scenario forecasts because they exclude uncertainty in the assumed drivers
Finance asks for next quarter's demand under three fuel-price scenarios, each with an 80% interval.
mediumdocumented fact - Put a gap between training and test windows when features or labels span time
Features include 14-day rolling sums, or labels mature only after a booking window closes.
mediumdocumented fact - Choose expanding or sliding training windows by backtest, not by habit
A logistics volume series changed behaviour after a network redesign two years ago, and the model trains on all history.
mediumpractitioner heuristic - Guard MASE against zero denominators and state which naive scaling you used
MASE is computed across thousands of SKUs and some values come back infinite or enormous.
mediumdocumented fact - Treat intermittent demand forecasts as rates and use bias-corrected Croston-type methods
A spare-parts series is zero most weeks with occasional orders of 3 to 20 units, and exponential smoothing chases each spike.
mediumdocumented fact - Separate stockout zeros from true zero demand before modelling a retail series
A store-SKU series looks intermittent, and stock records show the item was frequently unavailable.
mediumpractitioner heuristic - Reconcile hierarchical forecasts so route, region and network numbers agree
Route-level and network-level demand forecasts are produced separately, and the routes do not add up to the network total.
mediumdocumented fact - Match the forecasting level of a hierarchy to the noise and the decision
Store-SKU daily sales are too noisy to forecast well, but replenishment decisions are made per store and SKU.
mediumpractitioner heuristic - Evaluate prediction intervals on coverage and width together
Two interval methods are compared, and the team picks the one with the highest coverage.
mediumdocumented fact - Check interval coverage by season and segment, not only overall
Overall coverage of a 90% demand interval is 89%, and the team declares it calibrated.
mediumpractitioner heuristic - Pick the forecast quantile from the ratio of under- and over-forecast costs
Operations asks for 'the forecast' to set meal loads per flight, where running out costs more than waste.
mediumdocumented fact - Score a quantile forecast with pinball loss at that quantile
A team forecasts the 0.9 quantile of demand and reports its MAPE against actuals.
mediumdocumented fact - Model each seasonal period explicitly in hourly or daily data
Hourly electricity load shows daily, weekly and annual cycles, and a model with one seasonal period leaves strong patterns in the residuals.
mediumdocumented fact - Handle weekly data's non-integer annual season with Fourier terms or STL
A weekly demand series is fitted with a 52-period seasonal model, and the holiday peaks drift by a week every few years.
mediumdocumented fact - Encode moving holidays as windows before and after, not as a single day flag
A Turkish domestic route forecast misses the travel surge around Eid holidays, which move about eleven days earlier each Gregorian year.
mediumdocumented fact - Represent a known level shift with a step regressor or a post-break training window
A competitor added about 25% more seats on three routes from a known date, and the route forecasts now run high.
mediumdocumented fact - Bias-adjust back-transformed forecasts when totals or means matter
Daily demand is modelled on a log or Box-Cox scale, and the back-transformed daily forecasts are summed into a monthly budget.
mediumdocumented fact - Enforce positivity with a transformation, not by clipping negative forecasts to zero
A linear or ML forecast of low-volume sales produces negative values, and the pipeline replaces them with zero.
mediumdocumented fact - Difference only as much as a unit-root test and backtest support
A model is fitted on second differences because the first-differenced plot 'still looked wavy'.
mediumdocumented fact - Read ACF and PACF as hints, then choose orders by information criteria and residual checks
An analyst picks ARIMA(2,1,1) because the PACF 'cuts off at lag 2'.
mediumpractitioner heuristic - Use bookings already on hand as a known-at-origin input for departure-level forecasts
A revenue-management team forecasts final bookings for a departure 21 days out using only past departures.
mediumempirical finding - Detrend or difference the target before using tree models on trending series
A gradient-boosted forecast of a growing subscription base flattens at the highest level seen in training.
mediumdocumented fact - Test one global model across many related series before maintaining thousands of local ones
A team fits and monitors a separate ARIMA for each of 4,000 store-SKU series.
mediumempirical finding - Scale each series by its own recent level when training a global model
A global model across routes of 50 to 5,000 weekly passengers is accurate on big routes and poor on small ones.
mediumpractitioner heuristic - Choose recursive, direct or multi-output strategies by backtest at your horizons
A one-step-ahead ML model is fed its own predictions to produce a 28-day forecast.
mediumempirical finding - Backtest recursive forecasts with their own predicted lags, as production will run
A recursive LightGBM forecast reports similar error at day 1 and day 14.
mediumpractitioner heuristic