Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Discount significant wins from underpowered tests because their effect sizes are exaggerated
A small or short test returns a statistically significant uplift that is several times larger than past changes of the same kind.
harddocumented fact - Only segment experiment results by attributes the treatment cannot change
A readout splits users by "saw the new fare-comparison widget", "completed search" or another behaviour measured after assignment.
harddocumented fact - Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Do not filter or control on a variable caused by both the exposure and the outcome
An analyst studies only customers who filed a complaint, or only flights that sold out, and finds a surprising association between two drivers.
harddocumented fact - Treat a clean pre-trend plot as weak support for parallel trends, not proof
A difference-in-differences analysis of a new airport lounge on one hub shows no significant pre-period difference in trends against other hubs, and the report calls parallel trends "confirmed".
harddocumented fact - With staggered rollouts, do not rely on a single two-way fixed-effects DiD coefficient
A self-service kiosk was rolled out to airports at different months, and the team estimates one coefficient with airport and month fixed effects.
harddocumented fact - Show the synthetic control's pre-period fit and a backdated placebo before showing the effect
A synthetic control built from other airports estimates that a new hub incentive raised passenger numbers, and the deck shows only the post-launch gap.
harddocumented fact - Curate a small, relevant donor pool for synthetic control instead of including every available unit
A synthetic control for one route uses all 200 other routes as donors and only 12 months of pre-period data.
harddocumented fact - Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
harddocumented fact - Separate prevalence changes from model decay before retraining on a production precision drop
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
hardpractitioner heuristic - Run a fixed leakage playbook when validation looks too good
A first model beats the incumbent by a wide margin, or one feature dominates importance.
hardpractitioner heuristic - Score the full pipeline on shuffled labels to catch leakage the code review missed
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
hardpractitioner heuristic - Report importance as a range across equally good models, not one model's ranking
Several models with nearly identical validation scores rank features differently, and you must present the drivers.
harddocumented fact - Switch from PDP to ALE when the feature of interest is strongly correlated with others
You need the effect shape of a feature that moves with others, such as aircraft capacity and route distance, or price and season.
harddocumented fact - Check LightGBM's monotone constraint method before concluding constraints cost too much accuracy
Adding monotone constraints in LightGBM drops validation performance noticeably and the team wants to remove them.
harddocumented fact - Avoid building reports or monitors on the meaning of individual principal components
A dashboard tracks "PC1 = price sensitivity" over time, or a downstream model consumes PCA outputs that are refitted each month.
hardpractitioner heuristic - Do not choose intermittent-demand models by MAE because it rewards forecasting zero
On a mostly-zero series, an all-zero forecast wins the MAE comparison against every model.
harddocumented fact - Unconstrain sold-out flights before training a demand forecast
A route forecast is trained on passengers flown, and many peak departures hit capacity.
harddocumented fact - Check exchangeability before trusting split-conformal coverage on a time series
Split conformal intervals are calibrated on last year's residuals and promised to give 90% coverage.
harddocumented fact - Track realised interval coverage online and adjust the miscoverage level when it drifts
A deployed 90% interval for hourly call volumes covered 90% in backtests but 78% over the last month.
harddocumented fact - Decide explicitly how to treat a regime-change period in training and evaluation
Two years of history include an abrupt collapse and slow recovery in travel demand.
hardpractitioner heuristic - Do not answer a price what-if by changing the price input of a demand forecast
A manager asks what bookings would be if fares were cut 5%, and the analyst re-scores the forecasting model with lower fares.
harddocumented fact - Model spill and unconstrained demand before estimating gains from added capacity
A planner asks for passenger gains if the aircraft on a route changes from 160 to 180 seats.
hardpractitioner heuristic - Evaluate an intervention against a counterfactual forecast with control series
A loyalty campaign launched on one market, and leadership wants to know whether bookings rose because of it.
harddocumented fact - Count every strategy variant you tried before trusting the best backtest
A trading or pricing signal looks strong after trying many lookback windows, thresholds and universes on the same history.
harddocumented fact - Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Treat Monte Carlo dropout uncertainty as a signal to validate, not a calibrated interval
Stakeholders want an uncertainty band around a neural demand forecast and you consider keeping dropout on at inference and averaging many passes.
hardpractitioner heuristic - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Scale the loss and the schedule correctly when you accumulate gradients
You emulate a batch of 256 with 8 micro-batches of 32 because the full batch does not fit in memory.
hardpractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Estimate key-value cache memory before promising long contexts on self-hosted models
An ML engineer must serve a self-hosted model with long contexts to many concurrent users on a fixed GPU budget.
harddocumented fact - Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Decompose multi-hop questions instead of hoping one retrieval finds everything
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
hardpractitioner heuristic - Treat writes to long-term agent memory as untrusted input with provenance
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
hardpractitioner heuristic - Close exfiltration channels such as auto-loaded links and images in model output
The chat interface renders model output as Markdown, including images and links to arbitrary domains, and the model can see private data.
hardpractitioner heuristic - Keep the model that reads untrusted content away from tools and private data
One agent reads incoming customer emails, can query customer records, and can send messages externally.
harddocumented fact - Fix the plan before reading untrusted data when the task allows it
An agent must process external content, for example summarise today's supplier emails into a report, with a predictable set of actions.
harddocumented fact - Simulate the false-ship rate of a probability-to-beat-control stopping rule before using it daily
A team replaces p-values with "probability that B beats A" and ships as soon as it exceeds 95% on any daily check.
harddocumented fact - Randomise fare tests by departure or route when treated and control passengers draw on the same seats
A fare experiment splits sessions between a discounted and the standard fare on the same departures.
harddocumented fact - Leave a washout gap between switchback periods when a pricing or ranking change lingers
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
harddocumented fact - Validate a short-term proxy against the long-term outcome in past experiments before deciding with it
Rebooking within 30 days is used as a proxy for annual customer value and changes ship when the proxy rises.
harddocumented fact - Estimate who responds with sample-splitting methods and confirm the top segments in a fresh randomised holdout
After a loyalty-offer test, the team fits outcome on treatment times 60 covariates on the same data to find "who responds".
hardpractitioner heuristic - Estimate many route or segment effects with partial pooling instead of testing each one separately
A test is read out for 40 origin markets with a few thousand passengers each, and the analyst lists those with a raw p below 0.05, or those that survive a correction.
harddocumented fact - Expect the best of many arms to disappoint on rollout and re-estimate the winner in a fresh sample
A pricing-page test runs 10 variants, and the top one shows +3.1% and is shipped on that number.
harddocumented fact - Test for sorting around the cutoff before trusting a regression discontinuity estimate
A status tier starts at a points threshold (hypothetical: 25,000), and members just above are compared with those just below to estimate the effect of lounge access.
harddocumented fact - Test whether a difference-in-differences result survives a change of scale before reporting it
A difference-in-differences on passenger counts says treated routes grew less than controls, while the same data in logs says the opposite.
harddocumented fact - Report the first-stage strength of an instrument and expect a weak one to pull estimates toward the biased answer
A weather-disruption indicator (hypothetical) is used to instrument the fare a rebooked passenger paid, and the first-stage F-statistic is about 3.
harddocumented fact - Do not treat a very large opt-in sample as more trustworthy than a small random one
A post-trip rating prompt collects 60,000 voluntary responses out of 2 million passengers, and the average is presented with a tiny confidence interval.
harddocumented fact - Plan power for a ratio metric from numerator-denominator covariance by customer, not the spread across rows
A test targets revenue per booked passenger, and the sample size was computed from the standard deviation of the ratio across bookings.
harddocumented fact - Estimate your program's prior win rate before interpreting a single significant result
Suppose only about one idea in ten has delivered in your experimentation program, and a new test reaches p = 0.04.
harddocumented fact - Model a many-to-many relationship with a bridge table and state how weights prevent double counting
A booking can carry several promotion codes, and an analyst joins bookings to a promo column to report revenue by promotion.
harddocumented fact - Move fast-changing attributes into a mini-dimension instead of versioning the whole dimension
A customer dimension with millions of rows has a loyalty tier, risk band and credit-limit band that change monthly, and every change makes a type 2 copy of the whole row.
harddocumented fact - Store the zone name for a future local event, not the offset you know today
A schedule table stores departure times for next year as timestamps with a fixed offset computed at load time, and a government changes its daylight-saving rule.
harddocumented fact - Disambiguate repeated local times with an offset or fold before parsing wall-clock timestamps
A gate-system export has local timestamps without offsets, and on the night clocks go back the hour 01:00-02:00 appears twice.
harddocumented fact - Keep two clocks for corrected facts, when it was true and when you learned it
Finance restates last month's revenue after a refund batch is posted, and the dashboard owner overwrites the monthly figures; a model trained on last month's numbers can no longer be reproduced.
hardpractitioner heuristic - Set the late-data allowance from the measured arrival-delay distribution
A streaming job aggregates bookings by event time with an arbitrary 10-minute allowance, and the daily total disagrees with the nightly batch by a stable fraction of a percent.
hardpractitioner heuristic - Match near-duplicate entities with scored comparisons and a clerical-review band
Two passenger files should be merged, names and birth dates are keyed by hand, and exact-match deduplication leaves obvious duplicates while a loose rule merges different people.
harddocumented fact - Build target encodings in time order on time-stamped data, because random cross fitting borrows from the future
A table of time-stamped outcomes, such as no-shows by departure date, is target encoded with a standard k-fold cross fit and then judged on a later period.
hardpractitioner heuristic - Shrink a thin route's target encoding toward its origin mean before the global mean
Route-level target encoding is shrunk toward the overall rate, yet routes sharing an origin behave alike and new routes keep appearing with a handful of flights.
hardpractitioner heuristic - Bracket a suspected MNAR column with a delta adjustment instead of arguing about MAR versus MNAR
A decision-critical column, such as stated income or a survey score, may be missing because of its own value, and the team debates the mechanism from missingness plots.
harddocumented fact - Replay logged arrival times to mask training features the way serving will see them
A feature comes from a feed that lands hours after the event; the warehouse history is complete, but a live scoring request often lacks the value.
hardpractitioner heuristic - Include the outcome when imputing for an inferential model and leave it out of a deployed imputer
You must impute predictors, and the same dataset serves both a coefficient-interpretation study and a model to be deployed for scoring.
harddocumented fact - Treat a feature's time-to-live as a modelling choice, since it decides which training rows receive nulls
A feature store applies a time-to-live to a feature view, and training sets are built with point-in-time joins over entities with irregular update gaps.
harddocumented fact - Add a train-versus-live classifier to catch feature drift that per-feature tests miss
Every per-feature drift test is quiet, yet model quality is slipping, and the features are strongly related to one another.
hardpractitioner heuristic - Keep only features that survive repeated subsampling when selection is noisy
A lasso or importance-based selector returns a different shortlist when a few hundred rows change, and the team has to freeze a feature set.
harddocumented fact - Do not treat cross-validation fold scores as independent when judging a small model gap
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
harddocumented fact - Permute within strata when a feature is largely determined by a grouping variable
Permutation importance ranks route or season features high, and one of those features is almost fully set by the airport or month column.
hardempirical finding - Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
harddocumented fact - Expect ensemble variance reduction to stall at the correlation between members
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
harddocumented fact - Read min_child_weight as a sum of Hessians, not a row count, when moving from squared error to logistic loss
A minchildweight tuned for regression is reused for a binary:logistic model, or is expected to mean at least 20 rows.
harddocumented fact - Judge an interaction on the decision's scale: no log-odds interaction can still mean unequal probability gaps
A logistic model has two drivers, such as booking window and market, and the team asks whether they interact using the p-value of the product term.
hardpractitioner heuristic - Reach for a zero-inflated count model only when a separate process generates structural zeros
A count outcome has many zeros (lounge visits per passenger, complaints per station) and a Poisson fit looks poor.
harddocumented fact - Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
hardpractitioner heuristic - Validate that ensemble disagreement predicts error before routing cases to review on it
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
hardpractitioner heuristic - Estimate reconciliation weights and aggregate features only from data before each origin
A hierarchical forecast reconciles with MinT; its error covariance is estimated once from residuals over the whole history and then used in every backtest origin.
hardpractitioner heuristic - Treat hits of overlapping multi-step intervals as dependent when testing coverage
Intervals for a 14-day horizon are issued every day, and coverage is computed over 120 consecutive origins as if each hit were independent.
harddocumented fact - Evaluate a demand forecast against sales through the capacity cap, using quantiles
A model forecasts demand and is scored against recorded bookings on flights that sold out; bookings are capped at 180 seats.
hardpractitioner heuristic - Do not measure bias on uncensored days only, because dropping stock-outs selects low-demand days
To avoid censoring, a team computes forecast bias only on days that did not sell out and reports a persistent over-forecast of 10%.
hardpractitioner heuristic - Replay intermittent-demand forecasts through a stock policy and report service and stock, not only error
Spare-parts demand is mostly zeros with occasional orders, and two methods tie on MASE.
hardpractitioner heuristic - Compare two forecasts with a paired test on per-origin loss differences
Model B has a 3% lower mean error than model A over 52 weekly origins and is proposed as the replacement.
harddocumented fact - Log the action taken on each forecast so that monitored error is not contaminated by your own response
A capacity forecast triggers extra flights on routes forecast high, and the monitored error on those routes keeps improving.
hardpractitioner heuristic - Decide the ARIMA constant deliberately, because with differencing it sets the long-run trend
An automatic ARIMA run on quarterly fuel-cost data returns d = 1 with a non-zero constant, and the five-year forecast climbs in a straight line.
harddocumented fact - In a regression with ARIMA errors, test the ARIMA innovations and not the regression residuals
A model with a promotion regressor and ARIMA(1,0,2) errors is checked by running a Ljung-Box test on the regression residuals, and the test fails.
harddocumented fact - Train on the driver forecasts that existed at each origin, not on the values that were later observed
A day-ahead load model is trained on observed temperature, but in production it receives a weather forecast whose error grows with lead time.
hardpractitioner heuristic - Pick the MinT covariance estimator by the number of series relative to the history length
A retail group reconciles forecasts for 4,000 item-store series with 104 weeks of history, using MinT with the sample covariance of one-step forecast errors.
harddocumented fact - Weight a global model's loss by the weights of the metric that will judge it
A global boosted model for 30,000 SKU series is trained with unweighted squared error, but it will be judged on an error that is weighted by recent revenue.
hardpractitioner heuristic - Use a probability-updating method such as TSB when items can become obsolete
A wholesaler's slow items sometimes stop selling for good, yet a Croston forecast stays at the old demand rate for months because it only updates when a sale occurs.
harddocumented fact - Treat a closed fare class as censoring even when the cabin is not full
A hypothetical flight with 180 seats has sold 118 when its lowest fare class closes, and that class's booking curve goes flat from that day.
hardpractitioner heuristic - Handle sparse and flat series before sending them to a mean-scaled, tokenizing forecaster
A pretrained model that scales each series by the mean absolute value of its context is run on spare-part demand where one week in twenty has a sale of one unit.
harddocumented fact - Feed a statistical model's out-of-sample forecast to the ML model as a feature, rebuilt at every origin
A boosted-tree forecaster is worse than ETS at trend and seasonal shape but better at promotions, and the team wants both strengths.
hardpractitioner heuristic - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Roll back the model together with its preprocessing, schema and stored state
A model v7 is rolled back to v6 after a regression, but v7 introduced a new feature column, a new encoder file and updated entries in an online store.
hardpractitioner heuristic - Alert on error-budget burn rate over two windows for model service objectives
A model endpoint has a latency or valid-response objective, and the on-call is paged by a fixed threshold that either fires on every blip or misses a slow degradation.
harddocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Budget the tail latency of every dependency when one request fans out to many lookups
Each prediction needs 20 parallel feature lookups, and every store has a good median and a p99 of 80 ms.
hardpractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Evaluate against past decision-makers with a selective-labels method, not on approved cases alone
A credit model is scored on repaid versus defaulted loans, but outcomes exist only for applicants that underwriters approved (hypothetical).
hardpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic - Start residual blocks as the identity by zero-initialising their last scale
A deep residual network trains with a very large batch, or its loss is erratic during the first hundred steps even after warmup.
hardempirical finding - Measure steps to target at several batch sizes before paying for bigger batches
A team wants to move a training job from 4 to 32 accelerators and assumes the run will finish about 8 times sooner.
hardempirical finding - Estimate the gradient noise scale to see when a larger batch has stopped helping
Training is stable at your current batch size and you are deciding whether to double it, but a full sweep is too expensive.
hardempirical finding - Re-check calibration on shifted data, because a temperature fitted in distribution may not travel
A temperature-scaled network is deployed to a new region, season or device mix and its probabilities feed a threshold chosen on the old validation split.
hardempirical finding - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Include Adam's epsilon in the search space before ranking optimizers
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
hardempirical finding - Embed numerical features before the MLP when a tabular network trails gradient boosting
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
hardempirical finding - Do not stop at the first rise in validation loss without checking a longer horizon
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
hardempirical finding - Centre or whiten sentence embeddings before nearest-neighbour search and compare with the raw vectors
Scores between unrelated texts from a BERT-based encoder are all high, and every query seems to retrieve similar neighbours.
hardpractitioner heuristic