Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Do not let a high ROC-AUC on rare events hide poor precision at the operating point
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
mediumempirical finding - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Treat a negative test R² as a signal of shift or broken features, not as a rounding curiosity
A regression model that scored well in validation shows negative R² on a new test period or segment.
mediumdocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Judge probability quality with log loss or Brier score, and calibration separately
Predicted probabilities will feed expected-value calculations, pricing or risk limits, and you only report ROC-AUC.
mediumdocumented fact - Check a reliability diagram before multiplying predicted probabilities by money
A boosted tree, random forest or SVM score is about to be used as a probability in an expected-cost or expected-revenue formula.
mediumempirical finding - Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
You choose a calibration method for a classifier with a modest validation set.
mediumdocumented fact - Recalibrate after resampling or class weighting, because both inflate predicted risk
You trained with SMOTE, undersampling, classweight='balanced', isunbalance or scaleposweight, and now use predictproba.
mediumempirical finding - Try the untouched data plus a cost-based threshold before class weights or resampling
Positives are rare and the instinct is to rebalance the training data.
mediumdocumented fact - Treat SMOTE as an experiment with known caveats, not a default fix for imbalance
Someone proposes SMOTE because the positive class is 3% of rows.
mediumempirical finding - Resample inside the cross-validation pipeline so validation folds keep the real class mix
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
easydocumented fact - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - Tune the threshold on cross-validated or fresh validation predictions, never on training predictions
You search thresholds to maximize a business metric such as profit or F-beta.
mediumdocumented fact - When reviewers can handle only K cases, rank and evaluate precision at K instead of choosing a probability cut-off
An operations team can investigate a fixed number of alerts per day regardless of how many the model flags.
mediumpractitioner heuristic - Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
harddocumented fact - Separate prevalence changes from model decay before retraining on a production precision drop
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
hardpractitioner heuristic - Report a tuned model's performance from nested cross-validation, not from the search's best score
You quote GridSearchCV.bestscore (or the best of many experiments) as the expected production performance.
mediumdocumented fact - Count every look at the test set as a selection step and refresh the holdout when looks pile up
A team checks the test score after each feature tweak, threshold change or model swap over several weeks.
mediumdocumented fact - Compare candidate models on identical folds and read per-fold differences before declaring a winner
Two models differ by a few thousandths in mean cross-validated score.
mediumpractitioner heuristic - Split by entity with grouped cross-validation when the same customer, aircraft or patient appears many times
Rows are repeated measurements of the same entity, and the model will score new entities in production.
mediumdocumented fact - Validate time-ordered tabular models on later periods, even when the task is not a forecast
A churn, credit or no-show model is trained on rows spanning two years and validated with shuffled K-fold.
mediumdocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Run a fixed leakage playbook when validation looks too good
A first model beats the incumbent by a wide margin, or one feature dominates importance.
hardpractitioner heuristic - Score the full pipeline on shuffled labels to catch leakage the code review missed
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
hardpractitioner heuristic - Plot a learning curve before asking for more data
Stakeholders ask whether another year of history or more labelling budget will improve the model.
easydocumented fact - Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
easydocumented fact - Give early stopping its own validation split and never point it at the test set
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
easydocumented fact - Know that HistGradientBoosting switches early stopping on automatically for larger datasets
A HistGradientBoosting model behaves differently on a full dataset than on a sample, or training sizes vary across experiments.
mediumdocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Do not read lasso's choice among correlated features as evidence that the others do not matter
Lasso keeps one of several correlated features (competitor price, fare level, season) and zeroes the rest; someone proposes dropping them from the data.
mediumdocumented fact - Interpret predictions, not individual coefficients, when features are strongly collinear
Coefficients change sign or size when a correlated feature is added, removed or the data period shifts.
mediumdocumented fact - Treat a funnel-shaped residual plot as a modelling decision, not cosmetic noise
Residuals versus fitted values widen as predictions grow, common for revenue, demand and claim size.
mediumdocumented fact - Plot residuals in time order even for models that are not forecasts
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
mediumdocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Compare candidate and incumbent per slice before shipping an aggregate improvement
A new model beats the production model on the overall metric and a release is planned.
mediumpractitioner heuristic - Read the most confident wrong predictions first when debugging a classifier
You start error analysis on a classifier with thousands of misclassifications.
easypractitioner heuristic - Refuse to turn SHAP values into intervention advice without a causal design
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
mediumdocumented fact - Group correlated features before reading SHAP rankings
Several features measure the same thing (fare level, average fare, revenue per passenger) and SHAP ranks them separately.
mediumpractitioner heuristic - State the output scale of SHAP values for boosted classifiers before presenting them
You show SHAP values from a gradient-boosted classifier as percentage-point changes in probability.
mediumdocumented fact - Cluster correlated features before trusting permutation importance
Permutation importance on a held-out set shows every feature in a correlated group as nearly useless.
mediumdocumented fact - Prefer held-out permutation importance to impurity importance for tree models
You report the featureimportances attribute of a random forest or gradient-boosting model.
mediumdocumented fact - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Report importance as a range across equally good models, not one model's ranking
Several models with nearly identical validation scores rank features differently, and you must present the drivers.
harddocumented fact - Plot ICE curves with the PDP so averages do not hide opposite effects
A partial dependence plot shows a flat or mild effect for a feature that domain experts expect to matter.
mediumdocumented fact - Switch from PDP to ALE when the feature of interest is strongly correlated with others
You need the effect shape of a feature that moves with others, such as aircraft capacity and route distance, or price and season.
harddocumented fact - Encode known directional relationships with monotonic constraints in gradient boosting
Domain logic requires a direction (higher debt never lowers default risk, more days to departure never lowers remaining bookings), yet partial dependence wiggles against it.
mediumdocumented fact - Check LightGBM's monotone constraint method before concluding constraints cost too much accuracy
Adding monotone constraints in LightGBM drops validation performance noticeably and the team wants to remove them.
harddocumented fact - Test whether clusters survive resampling before naming them
A k-means run on customer or route features produces k tidy segments that the business wants to name and target.
mediumdocumented fact - Do not use the silhouette score to choose between centroid and density-based clustering
You compare k-means and DBSCAN results using the silhouette coefficient and pick the higher score.
mediumdocumented fact - Validate segments on outcomes they were not built from before calling them personas
A clustering of customers or routes is about to become a set of named personas with tailored strategies.
mediumpractitioner heuristic - Do not assume the top principal components carry the predictive signal
You reduce features with PCA before a supervised model and keep components explaining 95% of variance.
mediumdocumented fact - Avoid building reports or monitors on the meaning of individual principal components
A dashboard tracks "PC1 = price sensitivity" over time, or a downstream model consumes PCA outputs that are refitted each month.
hardpractitioner heuristic - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Set the contamination parameter from review capacity and a labelled sample, not as a guess of the anomaly rate
You configure IsolationForest or Local Outlier Factor and must choose contamination.
mediumdocumented fact - Do not score pooled out-of-fold predictions as if they were one test set
You call crossvalpredict, compute one AUC or RMSE over all pooled predictions and report it as the cross-validated score.
easydocumented fact - Do not treat cross-validation fold scores as independent when judging a small model gap
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
harddocumented fact - Build learning curves by number of entities, not number of rows, when rows cluster
Each customer, aircraft or flight contributes dozens of rows and the learning curve is built by subsampling rows.
mediumpractitioner heuristic - Treat the out-of-bag score as optimistic when rows are grouped or time-ordered
A random forest reports oobscore and you plan to skip validation because the out-of-bag estimate is free.
mediumpractitioner heuristic - Set LightGBM min_data_in_leaf from how many rows you would trust to estimate a rate
A leaf-wise boosted model on a few thousand rows overfits and the usual response is to lower numleaves again and again.
mediumdocumented fact - Count tied scores before you read precision at K or NDCG from a shallow model
A shallow tree or a model with few distinct leaf values produces the scores, and precision at 100 is reported from a sort.
mediumdocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Let the stake of each case set its own action threshold when costs scale with value
A single probability cut-off is used for retention offers, refunds or fraud holds although the money at risk varies by a factor of fifty between cases.
mediumpractitioner heuristic - Track alert volume at a fixed score threshold, because score drift moves it silently
A model with a fixed 0.30 cut-off has run for months and operations complains that the queue is twice the planned size.
mediumpractitioner heuristic - Pick a threshold in the flat part of the expected-cost curve, not at its sharpest minimum
Your cost curve over thresholds has a minimum at 0.19 but the unit costs behind it are rough estimates.
mediumpractitioner heuristic - Check calibration inside each segment, because overall calibration can hide offsetting errors
The reliability diagram over all rows hugs the diagonal and the model is declared calibrated.
mediumpractitioner heuristic - Check ranking metrics after isotonic calibration, which turns scores into plateaus
You add isotonic calibration to a model whose top-K list or AUC is used downstream.
mediumdocumented fact - State the background data behind SHAP values, because it defines what a feature value is compared with
SHAP bars for a tree model are presented as how much each feature matters, with no mention of the reference rows.
mediumdocumented fact - Report permutation importance with its repeat spread and the metric that matches the decision
A permutation importance table shows one number per feature from a single shuffle, scored with the default metric.
easydocumented fact - Permute within strata when a feature is largely determined by a grouping variable
Permutation importance ranks route or season features high, and one of those features is almost fully set by the airport or month column.
hardempirical finding - Compare a model gap with the seed-to-seed spread before declaring a winner
Gradient boosting or forest candidates differ by 0.003 in validation AUC and each was trained once with a single seed.
easyempirical finding - Measure how often two labelers agree before treating a score gap as progress
Labels come from analyst judgement or agent coding and the team tunes models for another point of accuracy.
mediumpractitioner heuristic - Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
harddocumented fact - Lower the learning rate and let early stopping choose the tree count
A boosted model with learning rate 0.3 and 200 trees is tuned by hand on several other parameters first.
easydocumented fact - Expect ensemble variance reduction to stall at the correlation between members
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
harddocumented fact - Train a stacking meta-model on out-of-fold predictions, never on in-sample ones
A second-level model combines a boosted model, a forest and a linear model, and was fitted on their predictions for the training rows.
mediumdocumented fact - Retune C when class weights change, because weights rescale the data term against the penalty
You switch an SVM or logistic regression to classweight balanced and keep the C chosen without weights.
mediumdocumented fact - Treat huge logistic coefficients and unstable standard errors as a sign of separation
A logistic regression returns coefficients of 15 or 30 for a dummy variable, enormous standard errors or a convergence warning.
mediumdocumented fact - Choose the simplest regularization setting within one standard error of the best
Cross-validation picks the lowest-error alpha from a path of 100 values and the error curve is flat across a wide stretch.
mediumpractitioner heuristic - Plot the coefficient path before naming the drivers a lasso model selected
A lasso fit at one alpha keeps eight features and the report calls them the key drivers.
mediumdocumented fact - Judge a slice's error rate against sample-size control limits before calling it the problem
A slice table sorted by error rate puts a 20-row segment at the top with 25% errors while the model averages 10%.
mediumdocumented fact - Search scale hyperparameters on a log scale and prefer random sampling to a grid
A grid of C in {0.1, 0.5, 1, 5, 10} and alpha in {0.001, 0.002, 0.003} is searched for a model with six hyperparameters.
easyempirical finding - Guard tree-model inputs against values outside the training range, because predictions go flat instead of failing
A numeric feature at scoring time can exceed anything seen in training, such as a larger aircraft type or a new price tier.
mediumdocumented fact - Set a minimum leaf size when tree-model probabilities feed a ranking or a threshold
A single tree or a forest with fully grown leaves outputs probabilities that bunch at 0 and 1, and a review list or cut-off is built on them.
mediumdocumented fact - Read min_child_weight as a sum of Hessians, not a row count, when moving from squared error to logistic loss
A minchildweight tuned for regression is reused for a binary:logistic model, or is expected to mean at least 20 rows.
harddocumented fact - Unit-test monotone constraints with paired counterfactual rows instead of trusting the parameter
A boosted model carries monotone constraints (a higher fare must not raise predicted bookings) and is retrained often while columns, dtypes and library versions change.
mediumpractitioner heuristic - Check how pandas ordered categoricals reach LightGBM before assuming they are split as categories
A column typed as an ordered pandas categorical (fare class, risk grade) goes into LightGBM with categoricalfeature left at its default.
mediumdocumented fact - Model counts per exposure with a log-exposure offset instead of raw counts or an unweighted rate
Counts of no-shows, claims or defects accrue over very different amounts of exposure: flights flown, policy-years, machine-hours.
mediumdocumented fact - Read a convergence warning from a penalized linear model as a scaling problem before raising max_iter
A logistic regression warns that it did not converge after features in very different units (fares, counts, 0/1 flags) were added.
mediumdocumented fact - Judge an interaction on the decision's scale: no log-odds interaction can still mean unequal probability gaps
A logistic model has two drivers, such as booking window and market, and the team asks whether they interact using the p-value of the product term.
hardpractitioner heuristic - Compare Pearson chi-square to residual degrees of freedom before trusting Poisson standard errors
A Poisson regression for counts (no-shows per flight, tickets per shift) returns narrow intervals and very significant drivers.
mediumdocumented fact - Reach for a zero-inflated count model only when a separate process generates structural zeros
A count outcome has many zeros (lounge visits per passenger, complaints per station) and a Poisson fit looks poor.
harddocumented fact - Use a Poisson objective for count targets in boosted trees and judge it on Poisson deviance
Counts or frequencies (no-shows per flight, claims per policy) are predicted by a boosted tree with squared error.
mediumdocumented fact - Choose the number of clusters from the action you can take and a stability check, not from inertia
A segmentation run reports inertia for k from 2 to 15 and the team names segments at the apparent elbow.
mediumdocumented fact - Handle DBSCAN's noise label as an explicit population in downstream tables, not as a cluster or a discard
DBSCAN labels are joined back to a customer or sensor table and label -1 is used in group-bys, one-hot encoded as cluster -1, or dropped.
mediumdocumented fact - Cluster in the original feature space, not on a t-SNE map, and use the map only to look
A t-SNE picture of customer or booking features shows islands, and k-means is run on its two coordinates or segment sizes are read off the plot.
mediumempirical finding - Count the minority events in each validation fold before trusting a rare-event metric
An event with about 1% prevalence is validated with 5-fold cross-validation on 3,000 rows, which leaves about 30 events in total and 6 per fold.
mediumpractitioner heuristic - Publish per-class recall with macro and weighted averages when a multiclass problem has rare classes
A multiclass model (disruption cause, claim type, complaint category) is summarized by accuracy or weighted F1, and one costly class holds a few percent of rows.
easydocumented fact - Score anomalies against a peer group, not against the whole population
One detector (isolation forest, local outlier factor, z-score) is fitted on flights, machines or accounts that sit in very different regimes.
mediumpractitioner heuristic - Verify the sign convention of an anomaly score before ranking, thresholding or comparing detectors
Scores from several detectors (scoresamples, decisionfunction, negativeoutlierfactor) are ranked, averaged or thresholded in one pipeline.
easydocumented fact - Choose outlier or novelty mode from whether the training data may already contain anomalies
A local outlier factor model is fitted on historical records and then asked to score new records, or its predictions on the training rows are used to evaluate it.
mediumdocumented fact - Evaluate a ranker per list and average, instead of pooling all items into one AUC
A model orders items inside lists (offers for a user, fares for a search, cases for an analyst) and is scored with one AUC or precision across all pooled items.
mediumdocumented fact - Beat a popularity baseline and a repeat-previous baseline before crediting a recommender's personalization
A new recommendation model shows lift in offline hit rate over random or over the current page order.
mediumpractitioner heuristic - Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
hardpractitioner heuristic - Do not present the spread of tree predictions in a forest as a prediction interval
A random forest or bagged model is asked for uncertainty, and the standard deviation or 5th to 95th percentile of its tree predictions is shown as an interval.
mediumpractitioner heuristic - Validate that ensemble disagreement predicts error before routing cases to review on it
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
hardpractitioner heuristic - Document the label definition and the scored population in the model record, not only the features
A model for churn, no-show or default is handed over, audited or retrained by someone who was not in the original project.
easypractitioner heuristic - Publish a score contract that says whether the number is a probability, a rank or a margin, and how it was calibrated
Downstream teams consume a model's score in pricing, capacity or campaign logic that they own.
mediumpractitioner heuristic - Model an ordered target with ordered thresholds and score it by how many levels it misses
The target has ordered levels, such as complaint severity 1 to 5, and you are about to fit an unordered multiclass model and report accuracy.
mediumpractitioner heuristic - Fit a logit-link model to a bounded share such as load factor instead of a straight line
The target is a share between 0 and 1, such as load factor, and a linear regression predicts values below 0 or above 1.
mediumpractitioner heuristic - Correct a share estimated by counting classifier labels when the classifier is imperfect and prevalence moved
A report estimates the share of positive cases, such as cancellations, by counting a classifier's positive labels in a new period.
hardpractitioner heuristic - Weight validation folds by the input density ratio when the scoring population differs in its inputs
You will score a population whose inputs differ from the training mix, such as a new route network, while the input-to-outcome relationship should hold, and cross-validation looks fine.
hardpractitioner heuristic - Recalibrate an average of calibrated models, because averaging probabilities makes the blend underconfident
Models that each pass a reliability check are blended by averaging their probabilities, and the blend feeds an expected-value or threshold rule.
mediumpractitioner heuristic - Compute the prevalence and cost range where a thresholded classifier beats alerting on everyone or no one
A fixed-threshold classifier tuned at one prevalence and cost ratio will be reused where either differs, such as another route group.
hardpractitioner heuristic - Summarise a precision-recall curve with average precision, not the trapezoidal area, when scores are coarse
A shallow tree, banded scorecard or isotonic-calibrated model gives only a handful of distinct scores, and its PR-AUC is the trapezoidal area under the curve.
mediumpractitioner heuristic