Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Run A/A tests on a new experimentation pipeline before trusting its first A/B result
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
mediumpractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - Specify ROWS and a unique ORDER BY in running totals, because the default frame groups ties
A running total of bookings uses SUM(...) OVER (ORDER BY bookingdate) and several rows share the same date.
mediumdocumented fact - Densify dates before a ROWS-based rolling window, because "7 rows" is not "7 days" when days are missing
A 7-day rolling average of passengers uses AVG(...) OVER (PARTITION BY route ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) on a table with gaps.
mediumdocumented fact - Partition lag features by entity and shift them by the forecast horizon, not by one row
A training table adds LAG(passengers) OVER (ORDER BY date) as a feature for a model that forecasts 14 days ahead.
mediumpractitioner heuristic - Report business metrics by event time and pipeline health by processing time
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
mediumdocumented fact - Mark recent periods as provisional until late-arriving data has had time to land
Yesterday's bookings look 15% low on the morning dashboard, and someone escalates a demand drop.
mediumdocumented fact - Make every load and backfill idempotent so a rerun cannot double-count
A failed nightly job is rerun manually, or a month of history is backfilled after a logic fix.
mediumdocumented fact - Join facts to the dimension version that was valid at event time, not to current attributes
A route was reclassified from domestic to intl-short last year, and a revenue-by-market report joins all history to the current route table.
mediumdocumented fact - Keep facts whose dimension record has not arrived yet, using an explicit unknown member
Bookings for a newly opened route arrive before the route appears in the route dimension, and the reporting query uses an inner join.
mediumdocumented fact - Build training rows with as-of joins so each feature uses only values known at the cutoff
A training table joins each departure to the latest customer and route features in the warehouse.
mediumdocumented fact - Store instants in UTC and assign the business date with the local zone of where it happened
Bookings from several countries are aggregated to daily totals using the server's clock or a UTC date.
mediumdocumented fact - Expect 23-hour and 25-hour days in hourly data around daylight-saving changes
An hourly demand or call-volume series in local time shows a missing hour in spring and a doubled hour in autumn.
mediumdocumented fact - Agree a data contract at the producer boundary before building on a shared table
Your forecast or KPI starts consuming a table owned by another team, with no written agreement about it.
mediumpractitioner heuristic - Standardize predictors before ridge, lasso or elastic net so the penalty is unit-free
You fit a penalized linear or logistic model where features are measured in different units.
mediumdocumented fact - Make target encoding fold-safe with cross fitting; fit().transform() on training rows leaks
You replace a categorical feature such as route or sales channel with the mean target per category.
mediumdocumented fact - Shrink target encodings of rare categories toward the global mean
Many categories have only a handful of rows, for example new routes or long-tail products.
mediumdocumented fact - Consider CatBoost's ordered target statistics when categorical encoding is the main leakage risk
A tabular problem is dominated by high-cardinality categoricals and your hand-built target encoding keeps leaking or overfitting.
mediumdocumented fact - Decide in advance how a model treats categories that first appear after deployment
A categorical feature grows over time: new routes, new products, new store codes.
mediumdocumented fact - Regularize LightGBM's native categorical splits when categories are many and small
You pass a high-cardinality integer-coded categorical to LightGBM as categoricalfeature.
mediumdocumented fact - Add a missing indicator when the fact of missingness may carry signal
A feature is missing more often for one segment or outcome, such as revenue missing more on intl-long flights in the synthetic raw table.
mediumdocumented fact - Monitor missing-rate shifts because a changed form or feed silently rewrites learned missingness
A model uses missing indicators or native NaN routing, and an upstream form, API or ETL job changes.
mediumpractitioner heuristic - Cap polynomial expansion degree and regularize, because generated features grow combinatorially
You apply PolynomialFeatures of degree 3 or more to dozens of inputs to capture nonlinearity.
mediumpractitioner heuristic - When you log-transform a skewed target, correct the back-transformed predictions toward the mean
You model log(revenue) or log(demand) and exponentiate predictions to report totals.
mediumdocumented fact - Express time features relative to the decision moment, not as absolute dates
A model uses raw dates or timestamps as features, for example booking date or departure date as ordinal numbers.
mediumpractitioner heuristic - Compute entity history features as of the prediction timestamp with point-in-time joins
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
mediumdocumented fact - Run feature selection inside each training fold, never on the full dataset first
You screen hundreds of candidate features by correlation with the target or by model importance, then cross-validate the shortlist.
mediumdocumented fact - Normalize count and frequency encodings by window length so they survive changes in data volume
You encode a category by how often it appears, or an entity by its raw event count.
mediumpractitioner heuristic - Ship the fitted preprocessing with the model so training and serving transform rows identically
Preprocessing lives in a notebook while serving engineers reimplement it in another service or language.
mediumdocumented fact - Do not let a high ROC-AUC on rare events hide poor precision at the operating point
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
mediumempirical finding - Treat a negative test R² as a signal of shift or broken features, not as a rounding curiosity
A regression model that scored well in validation shows negative R² on a new test period or segment.
mediumdocumented fact - Judge probability quality with log loss or Brier score, and calibration separately
Predicted probabilities will feed expected-value calculations, pricing or risk limits, and you only report ROC-AUC.
mediumdocumented fact - Check a reliability diagram before multiplying predicted probabilities by money
A boosted tree, random forest or SVM score is about to be used as a probability in an expected-cost or expected-revenue formula.
mediumempirical finding - Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
You choose a calibration method for a classifier with a modest validation set.
mediumdocumented fact - Recalibrate after resampling or class weighting, because both inflate predicted risk
You trained with SMOTE, undersampling, classweight='balanced', isunbalance or scaleposweight, and now use predictproba.
mediumempirical finding - Try the untouched data plus a cost-based threshold before class weights or resampling
Positives are rare and the instinct is to rebalance the training data.
mediumdocumented fact - Treat SMOTE as an experiment with known caveats, not a default fix for imbalance
Someone proposes SMOTE because the positive class is 3% of rows.
mediumempirical finding - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - Tune the threshold on cross-validated or fresh validation predictions, never on training predictions
You search thresholds to maximize a business metric such as profit or F-beta.
mediumdocumented fact - Report a tuned model's performance from nested cross-validation, not from the search's best score
You quote GridSearchCV.bestscore (or the best of many experiments) as the expected production performance.
mediumdocumented fact - Count every look at the test set as a selection step and refresh the holdout when looks pile up
A team checks the test score after each feature tweak, threshold change or model swap over several weeks.
mediumdocumented fact - Compare candidate models on identical folds and read per-fold differences before declaring a winner
Two models differ by a few thousandths in mean cross-validated score.
mediumpractitioner heuristic - Split by entity with grouped cross-validation when the same customer, aircraft or patient appears many times
Rows are repeated measurements of the same entity, and the model will score new entities in production.
mediumdocumented fact - Validate time-ordered tabular models on later periods, even when the task is not a forecast
A churn, credit or no-show model is trained on rows spanning two years and validated with shuffled K-fold.
mediumdocumented fact - Know that HistGradientBoosting switches early stopping on automatically for larger datasets
A HistGradientBoosting model behaves differently on a full dataset than on a sample, or training sizes vary across experiments.
mediumdocumented fact - Compare candidate and incumbent per slice before shipping an aggregate improvement
A new model beats the production model on the overall metric and a release is planned.
mediumpractitioner heuristic - Group correlated features before reading SHAP rankings
Several features measure the same thing (fare level, average fare, revenue per passenger) and SHAP ranks them separately.
mediumpractitioner heuristic - State the output scale of SHAP values for boosted classifiers before presenting them
You show SHAP values from a gradient-boosted classifier as percentage-point changes in probability.
mediumdocumented fact - Cluster correlated features before trusting permutation importance
Permutation importance on a held-out set shows every feature in a correlated group as nearly useless.
mediumdocumented fact - Prefer held-out permutation importance to impurity importance for tree models
You report the featureimportances attribute of a random forest or gradient-boosting model.
mediumdocumented fact - Encode known directional relationships with monotonic constraints in gradient boosting
Domain logic requires a direction (higher debt never lowers default risk, more days to departure never lowers remaining bookings), yet partial dependence wiggles against it.
mediumdocumented fact - Do not assume the top principal components carry the predictive signal
You reduce features with PCA before a supervised model and keep components explaining 95% of variance.
mediumdocumented fact - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Set the contamination parameter from review capacity and a labelled sample, not as a guess of the anomaly rate
You configure IsolationForest or Local Outlier Factor and must choose contamination.
mediumdocumented fact - Drop or forecast any regressor that is not known at the forecast origin for the full horizon
A demand model uses yesterday's competitor fare as a feature, and the business needs forecasts 1 to 28 days ahead.
mediumdocumented fact - Report ex-ante accuracy, not ex-post, when a model uses future-unknown regressors
A team reports that adding temperature cut daily electricity-load forecast error by a third.
mediumdocumented fact - Put a gap between training and test windows when features or labels span time
Features include 14-day rolling sums, or labels mature only after a booking window closes.
mediumdocumented fact - Choose expanding or sliding training windows by backtest, not by habit
A logistics volume series changed behaviour after a network redesign two years ago, and the model trains on all history.
mediumpractitioner heuristic - Score a quantile forecast with pinball loss at that quantile
A team forecasts the 0.9 quantile of demand and reports its MAPE against actuals.
mediumdocumented fact - Detrend or difference the target before using tree models on trending series
A gradient-boosted forecast of a growing subscription base flattens at the highest level seen in training.
mediumdocumented fact - Test one global model across many related series before maintaining thousands of local ones
A team fits and monitors a separate ARIMA for each of 4,000 store-SKU series.
mediumempirical finding - Scale each series by its own recent level when training a global model
A global model across routes of 50 to 5,000 weekly passengers is accurate on big routes and poor on small ones.
mediumpractitioner heuristic - Choose recursive, direct or multi-output strategies by backtest at your horizons
A one-step-ahead ML model is fed its own predictions to produce a 28-day forecast.
mediumempirical finding - Backtest recursive forecasts with their own predicted lags, as production will run
A recursive LightGBM forecast reports similar error at day 1 and day 14.
mediumpractitioner heuristic - Treat public-benchmark scores of forecasting foundation models as possibly contaminated
A time-series foundation model is proposed because it tops a public leaderboard.
mediumdocumented fact - Make a foundation or deep forecaster beat seasonal naive and ETS on your backtest first
A team plans to replace its forecasting stack with a transformer-based model after a promising demo.
mediumempirical finding - Store first-reported and final actuals so forecast accuracy uses the right version
Revenue figures are updated days after the flight date, and accuracy reports are rerun against the latest table.
mediumpractitioner heuristic - Compute training and serving features from one shared definition
Training features are built in a SQL notebook, and the online service reimplements them in application code.
mediumpractitioner heuristic - Log served features and replay them to measure skew, not just predictions
Offline AUC was 0.82, and live performance on matured labels is clearly lower.
mediumpractitioner heuristic - Give every feature a freshness SLA and enforce it at serving time
A fraud model's 'transactions in the last hour' feature comes from a stream that sometimes lags by 40 minutes.
mediumpractitioner heuristic - Design a degraded mode for when upstream features are late or missing
The nightly feature pipeline fails, and the pricing model must still publish recommendations by 06:00.
mediumpractitioner heuristic - Enforce backward-compatible schema changes and upgrade consumers first
An event producer wants to add a field and drop an old one in the stream your features read from.
mediumdocumented fact - Deduplicate late-arriving corrections by business key and load time, explicitly
A flight-day feed sometimes sends the same row twice, the second with an updated revenue figure.
mediumpractitioner heuristic - Exclude immature labels when building training data for slow outcomes
A default model labels loans as 'not defaulted' if no default is recorded yet, including loans issued last month.
mediumpractitioner heuristic - Monitor leading indicators while labels are delayed, then confirm with matured labels
Fraud chargebacks arrive up to 90 days after the transaction, so model accuracy for this month is unknown.
mediumpractitioner heuristic - Page on model impact symptoms, and send per-feature drift tests to a dashboard
A drift monitor runs a statistical test on 300 features daily and pages on every p-value below 0.05.
mediumdocumented fact - Confirm that drift hurts performance before retraining on it
A drift report shows the age distribution of applicants shifted, and the team schedules an urgent retrain.
mediumpractitioner heuristic - Use shadow mode to test plumbing and agreement, not to prove business impact
A new ranking model runs in shadow for two weeks, scores agree 92% with production, and the team claims a revenue uplift.
mediumpractitioner heuristic - Canary a new model on a small traffic slice with pre-agreed rollback criteria
A retrained model is ready, and the plan is to switch all traffic at once on Monday.
mediumdocumented fact - Define reproducibility as recoverable inputs plus tolerances, not identical bits
An auditor asks you to reproduce last year's model, and retraining gives slightly different numbers.
mediumdocumented fact - Put data, feature and model-quality tests in CI, not only unit tests for code
The ML repository's CI runs Python unit tests, and a broken join still reached production.
mediumdocumented fact - Gate releases on a frozen golden set with tolerances sized to run-to-run noise
A model-quality CI check fails randomly about one run in five, and engineers start rerunning until it passes.
mediumpractitioner heuristic - Make every automated decision reconstructable from a decision record
A customer disputes a declined credit-limit increase from eight months ago, and the team must explain it.
mediumpractitioner heuristic - Minimise personal data in features and give training snapshots a retention period
A churn model's training set copies full customer records, including fields the model never uses, and snapshots are kept indefinitely.
mediumdocumented fact - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Price inference per thousand predictions before choosing GPU serving
A team proposes GPU instances to serve a gradient-boosting model that handles 50 requests per second.
mediumpractitioner heuristic - Compare managed and self-hosted ML platforms on total cost including people
Self-hosting an open-source serving stack looks cheaper than a managed service on the cloud bill.
mediumpractitioner heuristic - Set retraining cadence from a measured decay curve and the cost of each retrain
A model is retrained daily because the platform makes it easy, and nobody has checked whether daily retrains help.
mediumpractitioner heuristic - Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
mediumpractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Retire models whose lift over a simple rule no longer pays for their upkeep
A churn model built three years ago still runs, but recent evaluations show it barely beats a tenure-and-usage rule.
mediumopinion - Adopt a feature store when features are shared or served online, not by default
A team with two batch models plans to deploy a feature store as its first MLOps step.
mediumopinion - Limit and explain forecast churn between model versions
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
mediumpractitioner heuristic - Test that examples in a batch cannot influence each other's predictions
You wrote custom reshaping, pooling or attention code with view, transpose or reductions over several axes.
mediumpractitioner heuristic - Check how many parameters your embedding tables hold before adding ID features
You add learned embeddings for high-cardinality identifiers such as customer, store or SKU IDs to a neural model.
mediumpractitioner heuristic - Pick softmax or independent sigmoids by asking whether labels can co-occur
You classify flight disruption reports into causes (weather, crew, technical, ATC) and some reports cite several causes.
mediumdocumented fact - Leave regression outputs linear and choose the loss for the statistic you need
A network predicts a non-negative quantity such as daily energy load or claim cost, and someone adds a ReLU on the output.
mediumpractitioner heuristic - Calibrate a neural classifier before using its softmax output as a probability
A network's predicted probability will drive a cost-based decision, such as how much overbooking risk to accept or when to route a claim to manual review.
mediumempirical finding - Match weight initialization to the activation function
A deep MLP or CNN with ReLU activations trains very slowly, or a tanh network's loss does not move for many steps.
mediumdocumented fact - Monitor the share of dead ReLU units after a learning-rate increase
You raised the learning rate to speed training, and accuracy then stalls below where it was at the lower rate.
mediumpractitioner heuristic - Run a learning-rate range test before tuning anything else
You start training a new architecture or dataset and are about to grid-search learning rate together with other hyperparameters.
mediumpractitioner heuristic - Retune the learning rate whenever you change the batch size
You move training to a bigger GPU and raise the batch size from 64 to 512 to finish faster, keeping the old learning rate.
mediumempirical finding - Use Adam to iterate fast, but test SGD with momentum before final training
Your model trained with Adam is close to final and the remaining question is test accuracy, not speed of iteration.
mediumempirical finding - With Adam, use decoupled weight decay (AdamW), not the weight_decay L2 term
You add regularization to an Adam-trained model by setting weightdecay in torch.optim.Adam.
mediumdocumented fact - Replace BatchNorm when memory forces very small batches
High-resolution inspection images or long sequences force a batch size of 2 to 8 per GPU in a CNN that uses BatchNorm.
mediumempirical finding - Early-stop neural training on the decision metric and restore the best checkpoint
A churn network uses early stopping on validation loss with default patience, and the final weights are whatever the last epoch produced.
mediumpractitioner heuristic - Gradient-check custom backward code in double precision
You implement a custom autograd function or a hand-written backward pass, for example for a custom pricing-aware loss.
mediumdocumented fact - Log the global gradient norm and clip it when spikes precede loss blow-ups
Training loss on a recurrent or deep model occasionally jumps by orders of magnitude, sometimes ending in NaN.
mediumdocumented fact - Diagnose vanishing gradients with per-layer gradient norms, then fix the architecture
A deep plain network trains no better than a shallower one, or its early layers barely change from initialization.
mediumdocumented fact - Triage a NaN loss in a fixed order, starting with the data
The loss of a fraud model becomes NaN in epoch 3, and the team starts changing the architecture.
mediumpractitioner heuristic - Use a gradient scaler with float16 mixed precision and compare against a float32 run
You enable float16 autocast to cut memory and speed up training of an image or text model.
mediumdocumented fact - Unscale gradients before clipping when training with a gradient scaler
You add gradient clipping to a training loop that already uses GradScaler for mixed precision.
mediumdocumented fact - Seed every library inside each data-loader worker
Augmentations draw randomness from NumPy, Python random or another library inside a Dataset, and the DataLoader runs several workers.
mediumdocumented fact - Split by source item before augmenting, never after
You multiply a small labelled image set, such as photos of damaged baggage or defect images, with crops and flips, then split the enlarged set randomly.
mediumpractitioner heuristic - Give the baseline and the neural model the same tuning budget
A comparison report shows a neural model beating logistic regression and boosting on a churn dataset.
mediumpractitioner heuristic - Report results over several seeds before claiming a neural improvement
A fine-tuned text classifier scores 1.2 points higher than the current one on a single run.
mediumempirical finding - Switch on deterministic GPU mode while bisecting a training regression
After a refactor, validation accuracy of a credit-risk network drops by half a point, and you want to find the commit responsible.
mediumdocumented fact - Watch the flatten-to-dense layer, not the convolutions, for parameter blow-up
A small CNN for shelf images or X-ray triage has far more parameters than expected and overfits quickly.
mediumdocumented fact - Start transfer learning with a linear probe, then fine-tune if it pays
You have a few thousand labelled examples and a pretrained backbone, and must decide whether to freeze it or fine-tune it.
mediumempirical finding