Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Use a pre-period covariate (CUPED-style adjustment) to shrink variance before buying more traffic
An experiment on revenue per user needs more weeks than the roadmap allows, and the metric has a strong user-level history.
hardempirical finding - Match the variance formula to the randomization unit when metrics are computed per page or per session
Users are randomized, but the metric is click-through per page view or conversion per search, and the analysis treats each page as an independent observation.
harddocumented fact - Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
harddocumented fact - Separate prevalence changes from model decay before retraining on a production precision drop
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
hardpractitioner heuristic - Run a fixed leakage playbook when validation looks too good
A first model beats the incumbent by a wide margin, or one feature dominates importance.
hardpractitioner heuristic - Score the full pipeline on shuffled labels to catch leakage the code review missed
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
hardpractitioner heuristic - Check LightGBM's monotone constraint method before concluding constraints cost too much accuracy
Adding monotone constraints in LightGBM drops validation performance noticeably and the team wants to remove them.
harddocumented fact - Avoid building reports or monitors on the meaning of individual principal components
A dashboard tracks "PC1 = price sensitivity" over time, or a downstream model consumes PCA outputs that are refitted each month.
hardpractitioner heuristic - Check exchangeability before trusting split-conformal coverage on a time series
Split conformal intervals are calibrated on last year's residuals and promised to give 90% coverage.
harddocumented fact - Track realised interval coverage online and adjust the miscoverage level when it drifts
A deployed 90% interval for hourly call volumes covered 90% in backtests but 78% over the last month.
harddocumented fact - Decide explicitly how to treat a regime-change period in training and evaluation
Two years of history include an abrupt collapse and slow recovery in travel demand.
hardpractitioner heuristic - Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Exclude biases and normalization scales from weight decay
You apply AdamW to a model with LayerNorm or BatchNorm by passing model.parameters() with one weight-decay value.
hardpractitioner heuristic - When a Transformer diverges early, check normalization placement and warmup
A Transformer trained from scratch shows loss spikes or NaNs in the first few hundred steps at a learning rate that works later in training.
hardempirical finding - Treat Monte Carlo dropout uncertainty as a signal to validate, not a calibrated interval
Stakeholders want an uncertainty band around a neural demand forecast and you consider keeping dropout on at inference and averaging many passes.
hardpractitioner heuristic - When BERT-style fine-tuning fails on some seeds, train longer with a smaller learning rate
Fine-tuning an encoder on a small classification set succeeds on some seeds and collapses to majority-class predictions on others.
hardempirical finding - Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
hardempirical finding - Scale the loss and the schedule correctly when you accumulate gradients
You emulate a batch of 256 with 8 micro-batches of 32 because the full batch does not fit in memory.
hardpractitioner heuristic - Use label smoothing knowing it changes calibration and distillation
You turn on label smoothing for a classifier that will later serve as the teacher for a smaller distilled model.
hardempirical finding - Estimate key-value cache memory before promising long contexts on self-hosted models
An ML engineer must serve a self-hosted model with long contexts to many concurrent users on a fixed GPU budget.
harddocumented fact - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Decompose multi-hop questions instead of hoping one retrieval finds everything
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
hardpractitioner heuristic - Treat writes to long-term agent memory as untrusted input with provenance
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
hardpractitioner heuristic - Close exfiltration channels such as auto-loaded links and images in model output
The chat interface renders model output as Markdown, including images and links to arbitrary domains, and the model can see private data.
hardpractitioner heuristic - Keep the model that reads untrusted content away from tools and private data
One agent reads incoming customer emails, can query customer records, and can send messages externally.
harddocumented fact - Fix the plan before reading untrusted data when the task allows it
An agent must process external content, for example summarise today's supplier emails into a report, with a predictable set of actions.
harddocumented fact - Leave a washout gap between switchback periods when a pricing or ranking change lingers
A fare-ranking algorithm is global, so the team alternates it on and off by hour and compares the two states.
harddocumented fact - Expect the best of many arms to disappoint on rollout and re-estimate the winner in a fresh sample
A pricing-page test runs 10 variants, and the top one shows +3.1% and is shipped on that number.
harddocumented fact - Store the zone name for a future local event, not the offset you know today
A schedule table stores departure times for next year as timestamps with a fixed offset computed at load time, and a government changes its daylight-saving rule.
harddocumented fact - Disambiguate repeated local times with an offset or fold before parsing wall-clock timestamps
A gate-system export has local timestamps without offsets, and on the night clocks go back the hour 01:00-02:00 appears twice.
harddocumented fact - Keep two clocks for corrected facts, when it was true and when you learned it
Finance restates last month's revenue after a refund batch is posted, and the dashboard owner overwrites the monthly figures; a model trained on last month's numbers can no longer be reproduced.
hardpractitioner heuristic - Set the late-data allowance from the measured arrival-delay distribution
A streaming job aggregates bookings by event time with an arbitrary 10-minute allowance, and the daily total disagrees with the nightly batch by a stable fraction of a percent.
hardpractitioner heuristic - Build target encodings in time order on time-stamped data, because random cross fitting borrows from the future
A table of time-stamped outcomes, such as no-shows by departure date, is target encoded with a standard k-fold cross fit and then judged on a later period.
hardpractitioner heuristic - Replay logged arrival times to mask training features the way serving will see them
A feature comes from a feed that lands hours after the event; the warehouse history is complete, but a live scoring request often lacks the value.
hardpractitioner heuristic - Treat a feature's time-to-live as a modelling choice, since it decides which training rows receive nulls
A feature store applies a time-to-live to a feature view, and training sets are built with point-in-time joins over entities with irregular update gaps.
harddocumented fact - Add a train-versus-live classifier to catch feature drift that per-feature tests miss
Every per-feature drift test is quiet, yet model quality is slipping, and the features are strongly related to one another.
hardpractitioner heuristic - Do not treat cross-validation fold scores as independent when judging a small model gap
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
harddocumented fact - Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
harddocumented fact - Expect ensemble variance reduction to stall at the correlation between members
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
harddocumented fact - Read min_child_weight as a sum of Hessians, not a row count, when moving from squared error to logistic loss
A minchildweight tuned for regression is reused for a binary:logistic model, or is expected to mean at least 20 rows.
harddocumented fact - Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
hardpractitioner heuristic - Validate that ensemble disagreement predicts error before routing cases to review on it
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
hardpractitioner heuristic - Log the action taken on each forecast so that monitored error is not contaminated by your own response
A capacity forecast triggers extra flights on routes forecast high, and the monitored error on those routes keeps improving.
hardpractitioner heuristic - Decide the ARIMA constant deliberately, because with differencing it sets the long-run trend
An automatic ARIMA run on quarterly fuel-cost data returns d = 1 with a non-zero constant, and the five-year forecast climbs in a straight line.
harddocumented fact - Train on the driver forecasts that existed at each origin, not on the values that were later observed
A day-ahead load model is trained on observed temperature, but in production it receives a weather forecast whose error grows with lead time.
hardpractitioner heuristic - Pick the MinT covariance estimator by the number of series relative to the history length
A retail group reconciles forecasts for 4,000 item-store series with 104 weeks of history, using MinT with the sample covariance of one-step forecast errors.
harddocumented fact - Weight a global model's loss by the weights of the metric that will judge it
A global boosted model for 30,000 SKU series is trained with unweighted squared error, but it will be judged on an error that is weighted by recent revenue.
hardpractitioner heuristic - Use a probability-updating method such as TSB when items can become obsolete
A wholesaler's slow items sometimes stop selling for good, yet a Croston forecast stays at the old demand rate for months because it only updates when a sale occurs.
harddocumented fact - Handle sparse and flat series before sending them to a mean-scaled, tokenizing forecaster
A pretrained model that scales each series by the mean absolute value of its context is run on spare-part demand where one week in twenty has a sale of one unit.
harddocumented fact - Feed a statistical model's out-of-sample forecast to the ML model as a feature, rebuilt at every origin
A boosted-tree forecaster is worse than ETS at trend and seasonal shape but better at promotions, and the team wants both strengths.
hardpractitioner heuristic - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Roll back the model together with its preprocessing, schema and stored state
A model v7 is rolled back to v6 after a regression, but v7 introduced a new feature column, a new encoder file and updated entries in an online store.
hardpractitioner heuristic - Alert on error-budget burn rate over two windows for model service objectives
A model endpoint has a latency or valid-response objective, and the on-call is paged by a fixed threshold that either fires on every blip or misses a slow degradation.
harddocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Budget the tail latency of every dependency when one request fans out to many lookups
Each prediction needs 20 parallel feature lookups, and every store has a good median and a p99 of 80 ms.
hardpractitioner heuristic - Start residual blocks as the identity by zero-initialising their last scale
A deep residual network trains with a very large batch, or its loss is erratic during the first hundred steps even after warmup.
hardempirical finding - Measure steps to target at several batch sizes before paying for bigger batches
A team wants to move a training job from 4 to 32 accelerators and assumes the run will finish about 8 times sooner.
hardempirical finding - Estimate the gradient noise scale to see when a larger batch has stopped helping
Training is stable at your current batch size and you are deciding whether to double it, but a full sweep is too expensive.
hardempirical finding - Re-check calibration on shifted data, because a temperature fitted in distribution may not travel
A temperature-scaled network is deployed to a new region, season or device mix and its probabilities feed a threshold chosen on the old validation split.
hardempirical finding - Interpolate fine-tuned and original weights to keep robustness a fine-tune would erase
Fine-tuning a strong pretrained model lifts accuracy on your validation split, but the model is used on inputs that look different from the fine-tuning data.
hardempirical finding - Include Adam's epsilon in the search space before ranking optimizers
A comparison of SGD with momentum and Adam is run with default settings, and Adam loses by a point on a tabular-style network.
hardempirical finding - Embed numerical features before the MLP when a tabular network trails gradient boosting
A plain MLP on raw standardised numerical features of a tabular problem (fare, days to departure, load factor) underperforms a gradient-boosted model by a clear margin.
hardempirical finding - Do not stop at the first rise in validation loss without checking a longer horizon
An over-parameterised network's validation error rises in mid-training, then, with patience set to five epochs, training halts and a smaller model is chosen instead.
hardempirical finding - Count how often each item shows up in top-k lists to catch hub documents in embedding search
In a vector search over support articles, the same handful of articles appears in the top 10 for many unrelated queries.
hardpractitioner heuristic - Reweight to the production class prior when labelling effort was split evenly across intents
A labelling budget was split equally over six intents so each has the same count, while production traffic has one dominant intent and several rare ones.
harddocumented fact - Test on sequences longer than any seen in training before assuming the position scheme extends
A model trained on sessions of up to 128 events will see sessions of 400 events in production.
hardempirical finding - Detach the hidden state between truncated-BPTT chunks and treat chunk length as the learnable dependency horizon
You train an RNN on one long stream by carrying the hidden state across consecutive chunks.
harddocumented fact - Do not assume a CNN uses its full theoretical receptive field
You size a CNN on paper so its theoretical receptive field covers the whole image or sensor window and conclude the context is covered.
hardempirical finding - Have the model quote the relevant evidence before it answers from a long input
Retrieval is perfect in a 40,000-token prompt, yet multi-step answers over the supplied evidence are worse than on a trimmed version of the same prompt.
hardempirical finding - Re-check preconditions at write time with a version token, not only at read time
An agent reads seat availability or a fare early in a long run and makes the booking change many steps later.
harddocumented fact - Record the undo step for every write and place irreversible steps last
An agent must perform a sequence of writes, such as hold a seat, charge a fee and send a confirmation, and a later step may fail.
harddocumented fact - Run a new write-capable agent in shadow mode and compare its proposals with human actions first
A team wants to give a drafting agent permission to apply refunds or rebookings after good offline test results.
hardpractitioner heuristic - Assert required steps and forbidden actions in a trajectory instead of matching an exact path
An agent that issues refunds reaches the right final state, but sometimes skips the identity check the policy requires.
hardpractitioner heuristic - Report code-generation pass@k from n samples with the unbiased estimator, at the k users actually get
A code or SQL assistant is scored by "any of k samples passes the tests", using k samples per task exactly.
harddocumented fact - Gate on a non-inferiority margin for the paired difference, not on a raw score threshold
A CI gate fails the build when the candidate scores below the baseline by any amount on a 400-item set.
hardpractitioner heuristic - Control who can write to the corpus, because a few planted passages can steer answers
A RAG assistant indexes a wiki, a ticket system and a shared drive that many employees and contractors can edit.
hardpractitioner heuristic - Restrict tool access for the rest of a session once untrusted text has entered the context
One long agent session reads external content such as email, web pages or tickets and also holds state-changing tools.
hardpractitioner heuristic - Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
hardpractitioner heuristic - Do not extrapolate small-model 8-bit results to large models, because outlier features break naive quantisation
A model that quantised cleanly at a few billion parameters is scheduled for the same naive 8-bit recipe at a much larger size.
hardempirical finding - Reserve a small random slice of routed traffic for the other model so you can measure what routing hides
A router sends easy requests to a small model and the team reports quality from the traffic the router chose.
hardpractitioner heuristic - Isolate long-prompt traffic or chunk prefill when document requests stall chat token streams
A shared deployment serves short chat turns and occasional 50,000-token document analyses, and chat users report streams that freeze for seconds.
hardempirical finding - Size the fallback model for the whole failover load and shed low-priority traffic first
The primary provider degrades and the fallback chain sends 100% of traffic to a secondary model whose quota was set for occasional use.
hardpractitioner heuristic - Correct a share estimated by counting classifier labels when the classifier is imperfect and prevalence moved
A report estimates the share of positive cases, such as cancellations, by counting a classifier's positive labels in a new period.
hardpractitioner heuristic - Weight validation folds by the input density ratio when the scoring population differs in its inputs
You will score a population whose inputs differ from the training mix, such as a new route network, while the input-to-outcome relationship should hold, and cross-validation looks fine.
hardpractitioner heuristic - Compute the prevalence and cost range where a thresholded classifier beats alerting on everyone or no one
A fixed-threshold classifier tuned at one prevalence and cost ratio will be reused where either differs, such as another route group.
hardpractitioner heuristic