Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Treat a clean pre-trend plot as weak support for parallel trends, not proof
A difference-in-differences analysis of a new airport lounge on one hub shows no significant pre-period difference in trends against other hubs, and the report calls parallel trends "confirmed".
harddocumented fact - With staggered rollouts, do not rely on a single two-way fixed-effects DiD coefficient
A self-service kiosk was rolled out to airports at different months, and the team estimates one coefficient with airport and month fixed effects.
harddocumented fact - Show the synthetic control's pre-period fit and a backdated placebo before showing the effect
A synthetic control built from other airports estimates that a new hub incentive raised passenger numbers, and the deck shows only the post-launch gap.
harddocumented fact - Curate a small, relevant donor pool for synthetic control instead of including every available unit
A synthetic control for one route uses all 200 other routes as donors and only 12 months of pre-period data.
harddocumented fact - Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
harddocumented fact - Separate prevalence changes from model decay before retraining on a production precision drop
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
hardpractitioner heuristic - Run a fixed leakage playbook when validation looks too good
A first model beats the incumbent by a wide margin, or one feature dominates importance.
hardpractitioner heuristic - Score the full pipeline on shuffled labels to catch leakage the code review missed
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
hardpractitioner heuristic - Report importance as a range across equally good models, not one model's ranking
Several models with nearly identical validation scores rank features differently, and you must present the drivers.
harddocumented fact - Switch from PDP to ALE when the feature of interest is strongly correlated with others
You need the effect shape of a feature that moves with others, such as aircraft capacity and route distance, or price and season.
harddocumented fact - Check LightGBM's monotone constraint method before concluding constraints cost too much accuracy
Adding monotone constraints in LightGBM drops validation performance noticeably and the team wants to remove them.
harddocumented fact - Avoid building reports or monitors on the meaning of individual principal components
A dashboard tracks "PC1 = price sensitivity" over time, or a downstream model consumes PCA outputs that are refitted each month.
hardpractitioner heuristic - Do not choose intermittent-demand models by MAE because it rewards forecasting zero
On a mostly-zero series, an all-zero forecast wins the MAE comparison against every model.
harddocumented fact - Unconstrain sold-out flights before training a demand forecast
A route forecast is trained on passengers flown, and many peak departures hit capacity.
harddocumented fact - Check exchangeability before trusting split-conformal coverage on a time series
Split conformal intervals are calibrated on last year's residuals and promised to give 90% coverage.
harddocumented fact - Track realised interval coverage online and adjust the miscoverage level when it drifts
A deployed 90% interval for hourly call volumes covered 90% in backtests but 78% over the last month.
harddocumented fact - Decide explicitly how to treat a regime-change period in training and evaluation
Two years of history include an abrupt collapse and slow recovery in travel demand.
hardpractitioner heuristic - Do not answer a price what-if by changing the price input of a demand forecast
A manager asks what bookings would be if fares were cut 5%, and the analyst re-scores the forecasting model with lower fares.
harddocumented fact - Model spill and unconstrained demand before estimating gains from added capacity
A planner asks for passenger gains if the aircraft on a route changes from 160 to 180 seats.
hardpractitioner heuristic - Evaluate an intervention against a counterfactual forecast with control series
A loyalty campaign launched on one market, and leadership wants to know whether bookings rose because of it.
harddocumented fact - Count every strategy variant you tried before trusting the best backtest
A trading or pricing signal looks strong after trying many lookback windows, thresholds and universes on the same history.
harddocumented fact - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Treat Monte Carlo dropout uncertainty as a signal to validate, not a calibrated interval
Stakeholders want an uncertainty band around a neural demand forecast and you consider keeping dropout on at inference and averaging many passes.
hardpractitioner heuristic - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Simulate the false-ship rate of a probability-to-beat-control stopping rule before using it daily
A team replaces p-values with "probability that B beats A" and ships as soon as it exceeds 95% on any daily check.
harddocumented fact - Randomise fare tests by departure or route when treated and control passengers draw on the same seats
A fare experiment splits sessions between a discounted and the standard fare on the same departures.
harddocumented fact - Leave a washout gap between switchback periods when a pricing or ranking change lingers
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
harddocumented fact - Validate a short-term proxy against the long-term outcome in past experiments before deciding with it
Rebooking within 30 days is used as a proxy for annual customer value and changes ship when the proxy rises.
harddocumented fact - Estimate who responds with sample-splitting methods and confirm the top segments in a fresh randomised holdout
After a loyalty-offer test, the team fits outcome on treatment times 60 covariates on the same data to find "who responds".
hardpractitioner heuristic - Estimate many route or segment effects with partial pooling instead of testing each one separately
A test is read out for 40 origin markets with a few thousand passengers each, and the analyst lists those with a raw p below 0.05, or those that survive a correction.
harddocumented fact - Expect the best of many arms to disappoint on rollout and re-estimate the winner in a fresh sample
A pricing-page test runs 10 variants, and the top one shows +3.1% and is shipped on that number.
harddocumented fact - Test for sorting around the cutoff before trusting a regression discontinuity estimate
A status tier starts at a points threshold (hypothetical: 25,000), and members just above are compared with those just below to estimate the effect of lounge access.
harddocumented fact - Test whether a difference-in-differences result survives a change of scale before reporting it
A difference-in-differences on passenger counts says treated routes grew less than controls, while the same data in logs says the opposite.
harddocumented fact - Report the first-stage strength of an instrument and expect a weak one to pull estimates toward the biased answer
A weather-disruption indicator (hypothetical) is used to instrument the fare a rebooked passenger paid, and the first-stage F-statistic is about 3.
harddocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Plan power for a ratio metric from numerator-denominator covariance by customer, not the spread across rows
A test targets revenue per booked passenger, and the sample size was computed from the standard deviation of the ratio across bookings.
harddocumented fact - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact - Model a many-to-many relationship with a bridge table and state how weights prevent double counting
A booking can carry several promotion codes, and an analyst joins bookings to a promo column to report revenue by promotion.
harddocumented fact - Move fast-changing attributes into a mini-dimension instead of versioning the whole dimension
A customer dimension with millions of rows has a loyalty tier, risk band and credit-limit band that change monthly, and every change makes a type 2 copy of the whole row.
harddocumented fact - Store the zone name for a future local event, not the offset you know today
A schedule table stores departure times for next year as timestamps with a fixed offset computed at load time, and a government changes its daylight-saving rule.
harddocumented fact - Disambiguate repeated local times with an offset or fold before parsing wall-clock timestamps
A gate-system export has local timestamps without offsets, and on the night clocks go back the hour 01:00-02:00 appears twice.
harddocumented fact - Keep two clocks for corrected facts, when it was true and when you learned it
Finance restates last month's revenue after a refund batch is posted, and the dashboard owner overwrites the monthly figures; a model trained on last month's numbers can no longer be reproduced.
hardpractitioner heuristic - Set the late-data allowance from the measured arrival-delay distribution
A streaming job aggregates bookings by event time with an arbitrary 10-minute allowance, and the daily total disagrees with the nightly batch by a stable fraction of a percent.
hardpractitioner heuristic - Match near-duplicate entities with scored comparisons and a clerical-review band
Two passenger files should be merged, names and birth dates are keyed by hand, and exact-match deduplication leaves obvious duplicates while a loose rule merges different people.
harddocumented fact - Build target encodings in time order on time-stamped data, because random cross fitting borrows from the future
A table of time-stamped outcomes, such as no-shows by departure date, is target encoded with a standard k-fold cross fit and then judged on a later period.
hardpractitioner heuristic - Shrink a thin route's target encoding toward its origin mean before the global mean
Route-level target encoding is shrunk toward the overall rate, yet routes sharing an origin behave alike and new routes keep appearing with a handful of flights.
hardpractitioner heuristic - Bracket a suspected MNAR column with a delta adjustment instead of arguing about MAR versus MNAR
A decision-critical column, such as stated income or a survey score, may be missing because of its own value, and the team debates the mechanism from missingness plots.
harddocumented fact - Replay logged arrival times to mask training features the way serving will see them
A feature comes from a feed that lands hours after the event; the warehouse history is complete, but a live scoring request often lacks the value.
hardpractitioner heuristic - Include the outcome when imputing for an inferential model and leave it out of a deployed imputer
You must impute predictors, and the same dataset serves both a coefficient-interpretation study and a model to be deployed for scoring.
harddocumented fact - Add a train-versus-live classifier to catch feature drift that per-feature tests miss
Every per-feature drift test is quiet, yet model quality is slipping, and the features are strongly related to one another.
hardpractitioner heuristic - Keep only features that survive repeated subsampling when selection is noisy
A lasso or importance-based selector returns a different shortlist when a few hundred rows change, and the team has to freeze a feature set.
harddocumented fact - Do not treat cross-validation fold scores as independent when judging a small model gap
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
harddocumented fact - Permute within strata when a feature is largely determined by a grouping variable
Permutation importance ranks route or season features high, and one of those features is almost fully set by the airport or month column.
hardempirical finding - Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
harddocumented fact - Expect ensemble variance reduction to stall at the correlation between members
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
harddocumented fact - Read min_child_weight as a sum of Hessians, not a row count, when moving from squared error to logistic loss
A minchildweight tuned for regression is reused for a binary:logistic model, or is expected to mean at least 20 rows.
harddocumented fact - Judge an interaction on the decision's scale: no log-odds interaction can still mean unequal probability gaps
A logistic model has two drivers, such as booking window and market, and the team asks whether they interact using the p-value of the product term.
hardpractitioner heuristic - Reach for a zero-inflated count model only when a separate process generates structural zeros
A count outcome has many zeros (lounge visits per passenger, complaints per station) and a Poisson fit looks poor.
harddocumented fact - Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
hardpractitioner heuristic - Validate that ensemble disagreement predicts error before routing cases to review on it
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
hardpractitioner heuristic - Estimate reconciliation weights and aggregate features only from data before each origin
A hierarchical forecast reconciles with MinT; its error covariance is estimated once from residuals over the whole history and then used in every backtest origin.
hardpractitioner heuristic - Treat hits of overlapping multi-step intervals as dependent when testing coverage
Intervals for a 14-day horizon are issued every day, and coverage is computed over 120 consecutive origins as if each hit were independent.
harddocumented fact - Evaluate a demand forecast against sales through the capacity cap, using quantiles
A model forecasts demand and is scored against recorded bookings on flights that sold out; bookings are capped at 180 seats.
hardpractitioner heuristic - Do not measure bias on uncensored days only, because dropping stock-outs selects low-demand days
To avoid censoring, a team computes forecast bias only on days that did not sell out and reports a persistent over-forecast of 10%.
hardpractitioner heuristic - Replay intermittent-demand forecasts through a stock policy and report service and stock, not only error
Spare-parts demand is mostly zeros with occasional orders, and two methods tie on MASE.
hardpractitioner heuristic - Compare two forecasts with a paired test on per-origin loss differences
Model B has a 3% lower mean error than model A over 52 weekly origins and is proposed as the replacement.
harddocumented fact - Log the action taken on each forecast so that monitored error is not contaminated by your own response
A capacity forecast triggers extra flights on routes forecast high, and the monitored error on those routes keeps improving.
hardpractitioner heuristic - Decide the ARIMA constant deliberately, because with differencing it sets the long-run trend
An automatic ARIMA run on quarterly fuel-cost data returns d = 1 with a non-zero constant, and the five-year forecast climbs in a straight line.
harddocumented fact - In a regression with ARIMA errors, test the ARIMA innovations and not the regression residuals
A model with a promotion regressor and ARIMA(1,0,2) errors is checked by running a Ljung-Box test on the regression residuals, and the test fails.
harddocumented fact - Train on the driver forecasts that existed at each origin, not on the values that were later observed
A day-ahead load model is trained on observed temperature, but in production it receives a weather forecast whose error grows with lead time.
hardpractitioner heuristic - Pick the MinT covariance estimator by the number of series relative to the history length
A retail group reconciles forecasts for 4,000 item-store series with 104 weeks of history, using MinT with the sample covariance of one-step forecast errors.
harddocumented fact - Weight a global model's loss by the weights of the metric that will judge it
A global boosted model for 30,000 SKU series is trained with unweighted squared error, but it will be judged on an error that is weighted by recent revenue.
hardpractitioner heuristic - Use a probability-updating method such as TSB when items can become obsolete
A wholesaler's slow items sometimes stop selling for good, yet a Croston forecast stays at the old demand rate for months because it only updates when a sale occurs.
harddocumented fact - Treat a closed fare class as censoring even when the cabin is not full
A hypothetical flight with 180 seats has sold 118 when its lowest fare class closes, and that class's booking curve goes flat from that day.
hardpractitioner heuristic - Handle sparse and flat series before sending them to a mean-scaled, tokenizing forecaster
A pretrained model that scales each series by the mean absolute value of its context is run on spare-part demand where one week in twenty has a sale of one unit.
harddocumented fact - Feed a statistical model's out-of-sample forecast to the ML model as a feature, rebuilt at every origin
A boosted-tree forecaster is worse than ETS at trend and seasonal shape but better at promotions, and the team wants both strengths.
hardpractitioner heuristic - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Evaluate against past decision-makers with a selective-labels method, not on approved cases alone
A credit model is scored on repaid versus defaulted loans, but outcomes exist only for applicants that underwriters approved (hypothetical).
hardpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic - Re-check calibration on shifted data, because a temperature fitted in distribution may not travel
A temperature-scaled network is deployed to a new region, season or device mix and its probabilities feed a threshold chosen on the old validation split.
hardempirical finding - Include Adam's epsilon in the search space before ranking optimizers
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
hardempirical finding - Embed numerical features before the MLP when a tabular network trails gradient boosting
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
hardempirical finding - Do not stop at the first rise in validation loss without checking a longer horizon
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
hardempirical finding - Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
hardpractitioner heuristic - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Count selection data as part of the budget when you only own a few dozen labels
A few-shot text classifier is reported at a score tuned on prompts and learning rates using a large extra validation set, though production will have only 60 labels.
hardempirical finding - Keep per-item annotator votes and test soft labels before collapsing them to a majority
Three annotators label each tone or intent example, the pipeline keeps the majority label, and a third of the items were split 2 to 1.
hardempirical finding - Reweight to the production class prior when labelling effort was split evenly across intents
A labelling budget was split equally over six intents so each has the same count, while production traffic has one dominant intent and several rare ones.
harddocumented fact - Detach the hidden state between truncated-BPTT chunks and treat chunk length as the learnable dependency horizon
You train an RNN on one long stream by carrying the hidden state across consecutive chunks.
harddocumented fact - Do not assume a CNN uses its full theoretical receptive field
You size a CNN on paper so its theoretical receptive field covers the whole image or sensor window and conclude the context is covered.
hardempirical finding - Run a new write-capable agent in shadow mode and compare its proposals with human actions first
A team wants to give a drafting agent permission to apply refunds or rebookings after good offline test results.
hardpractitioner heuristic - Assert required steps and forbidden actions in a trajectory instead of matching an exact path
An agent that issues refunds reaches the right final state, but sometimes skips the identity check the policy requires.
hardpractitioner heuristic - Debias a judge-scored pass rate with a small human-labelled sample instead of reporting it raw
A judge scores thousands of production conversations cheaply, but you also hold human labels for a small subset.
harddocumented fact - Rank many variants with a Bradley-Terry fit on sampled pairs rather than all-pairs win counts
You compare eight or more prompt or model variants by pairwise judging and the all-pairs table has dozens of cells with few items each.
harddocumented fact - Cluster the standard errors when eval questions share a document, ticket or customer
An eval has 300 questions written from 30 documents, ten per document, and you compute a score interval as if the 300 were independent.
hardempirical finding - Gate on a non-inferiority margin for the paired difference, not on a raw score threshold
A CI gate fails the build when the candidate scores below the baseline by any amount on a 400-item set.
hardpractitioner heuristic - Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
hardpractitioner heuristic - Correct a share estimated by counting classifier labels when the classifier is imperfect and prevalence moved
A report estimates the share of positive cases, such as cancellations, by counting a classifier's positive labels in a new period.
hardpractitioner heuristic - Weight validation folds by the input density ratio when the scoring population differs in its inputs
You will score a population whose inputs differ from the training mix, such as a new route network, while the input-to-outcome relationship should hold, and cross-validation looks fine.
hardpractitioner heuristic - Compute the prevalence and cost range where a thresholded classifier beats alerting on everyone or no one
A fixed-threshold classifier tuned at one prevalence and cost ratio will be reused where either differs, such as another route group.
hardpractitioner heuristic - Forecast a monthly series at several temporal aggregation levels and reconcile them
Monthly route demand is forecast with one model at one frequency, while planners also use quarterly and annual figures that disagree with the monthly sum.
hardempirical finding