Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Do not let a high ROC-AUC on rare events hide poor precision at the operating point
A model for a 0.5% event reports ROC-AUC above 0.9 and the team concludes it is ready.
mediumempirical finding - Treat a negative test R² as a signal of shift or broken features, not as a rounding curiosity
A regression model that scored well in validation shows negative R² on a new test period or segment.
mediumdocumented fact - Judge probability quality with log loss or Brier score, and calibration separately
Predicted probabilities will feed expected-value calculations, pricing or risk limits, and you only report ROC-AUC.
mediumdocumented fact - Check a reliability diagram before multiplying predicted probabilities by money
A boosted tree, random forest or SVM score is about to be used as a probability in an expected-cost or expected-revenue formula.
mediumempirical finding - Use sigmoid calibration on small calibration sets and isotonic only with plenty of data
You choose a calibration method for a classifier with a modest validation set.
mediumdocumented fact - Recalibrate after resampling or class weighting, because both inflate predicted risk
You trained with SMOTE, undersampling, classweight='balanced', isunbalance or scaleposweight, and now use predictproba.
mediumempirical finding - Try the untouched data plus a cost-based threshold before class weights or resampling
Positives are rare and the instinct is to rebalance the training data.
mediumdocumented fact - Treat SMOTE as an experiment with known caveats, not a default fix for imbalance
Someone proposes SMOTE because the positive class is 3% of rows.
mediumempirical finding - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - Tune the threshold on cross-validated or fresh validation predictions, never on training predictions
You search thresholds to maximize a business metric such as profit or F-beta.
mediumdocumented fact - Report a tuned model's performance from nested cross-validation, not from the search's best score
You quote GridSearchCV.bestscore (or the best of many experiments) as the expected production performance.
mediumdocumented fact - Count every look at the test set as a selection step and refresh the holdout when looks pile up
A team checks the test score after each feature tweak, threshold change or model swap over several weeks.
mediumdocumented fact - Compare candidate models on identical folds and read per-fold differences before declaring a winner
Two models differ by a few thousandths in mean cross-validated score.
mediumpractitioner heuristic - Split by entity with grouped cross-validation when the same customer, aircraft or patient appears many times
Rows are repeated measurements of the same entity, and the model will score new entities in production.
mediumdocumented fact - Validate time-ordered tabular models on later periods, even when the task is not a forecast
A churn, credit or no-show model is trained on rows spanning two years and validated with shuffled K-fold.
mediumdocumented fact - Know that HistGradientBoosting switches early stopping on automatically for larger datasets
A HistGradientBoosting model behaves differently on a full dataset than on a sample, or training sizes vary across experiments.
mediumdocumented fact - Compare candidate and incumbent per slice before shipping an aggregate improvement
A new model beats the production model on the overall metric and a release is planned.
mediumpractitioner heuristic - Group correlated features before reading SHAP rankings
Several features measure the same thing (fare level, average fare, revenue per passenger) and SHAP ranks them separately.
mediumpractitioner heuristic - State the output scale of SHAP values for boosted classifiers before presenting them
You show SHAP values from a gradient-boosted classifier as percentage-point changes in probability.
mediumdocumented fact - Cluster correlated features before trusting permutation importance
Permutation importance on a held-out set shows every feature in a correlated group as nearly useless.
mediumdocumented fact - Prefer held-out permutation importance to impurity importance for tree models
You report the featureimportances attribute of a random forest or gradient-boosting model.
mediumdocumented fact - Encode known directional relationships with monotonic constraints in gradient boosting
Domain logic requires a direction (higher debt never lowers default risk, more days to departure never lowers remaining bookings), yet partial dependence wiggles against it.
mediumdocumented fact - Do not assume the top principal components carry the predictive signal
You reduce features with PCA before a supervised model and keep components explaining 95% of variance.
mediumdocumented fact - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Set the contamination parameter from review capacity and a labelled sample, not as a guess of the anomaly rate
You configure IsolationForest or Local Outlier Factor and must choose contamination.
mediumdocumented fact - Build learning curves by number of entities, not number of rows, when rows cluster
Each customer, aircraft or flight contributes dozens of rows and the learning curve is built by subsampling rows.
mediumpractitioner heuristic - Treat the out-of-bag score as optimistic when rows are grouped or time-ordered
A random forest reports oobscore and you plan to skip validation because the out-of-bag estimate is free.
mediumpractitioner heuristic - Set LightGBM min_data_in_leaf from how many rows you would trust to estimate a rate
A leaf-wise boosted model on a few thousand rows overfits and the usual response is to lower numleaves again and again.
mediumdocumented fact - Count tied scores before you read precision at K or NDCG from a shallow model
A shallow tree or a model with few distinct leaf values produces the scores, and precision at 100 is reported from a sort.
mediumdocumented fact - Track alert volume at a fixed score threshold, because score drift moves it silently
A model with a fixed 0.30 cut-off has run for months and operations complains that the queue is twice the planned size.
mediumpractitioner heuristic - Pick a threshold in the flat part of the expected-cost curve, not at its sharpest minimum
Your cost curve over thresholds has a minimum at 0.19 but the unit costs behind it are rough estimates.
mediumpractitioner heuristic - Check calibration inside each segment, because overall calibration can hide offsetting errors
The reliability diagram over all rows hugs the diagonal and the model is declared calibrated.
mediumpractitioner heuristic - Check ranking metrics after isotonic calibration, which turns scores into plateaus
You add isotonic calibration to a model whose top-K list or AUC is used downstream.
mediumdocumented fact - State the background data behind SHAP values, because it defines what a feature value is compared with
SHAP bars for a tree model are presented as how much each feature matters, with no mention of the reference rows.
mediumdocumented fact - Train a stacking meta-model on out-of-fold predictions, never on in-sample ones
A second-level model combines a boosted model, a forest and a linear model, and was fitted on their predictions for the training rows.
mediumdocumented fact - Retune C when class weights change, because weights rescale the data term against the penalty
You switch an SVM or logistic regression to classweight balanced and keep the C chosen without weights.
mediumdocumented fact - Guard tree-model inputs against values outside the training range, because predictions go flat instead of failing
A numeric feature at scoring time can exceed anything seen in training, such as a larger aircraft type or a new price tier.
mediumdocumented fact - Set a minimum leaf size when tree-model probabilities feed a ranking or a threshold
A single tree or a forest with fully grown leaves outputs probabilities that bunch at 0 and 1, and a review list or cut-off is built on them.
mediumdocumented fact - Unit-test monotone constraints with paired counterfactual rows instead of trusting the parameter
A boosted model carries monotone constraints (a higher fare must not raise predicted bookings) and is retrained often while columns, dtypes and library versions change.
mediumpractitioner heuristic - Check how pandas ordered categoricals reach LightGBM before assuming they are split as categories
A column typed as an ordered pandas categorical (fare class, risk grade) goes into LightGBM with categoricalfeature left at its default.
mediumdocumented fact - Read a convergence warning from a penalized linear model as a scaling problem before raising max_iter
A logistic regression warns that it did not converge after features in very different units (fares, counts, 0/1 flags) were added.
mediumdocumented fact - Use a Poisson objective for count targets in boosted trees and judge it on Poisson deviance
Counts or frequencies (no-shows per flight, claims per policy) are predicted by a boosted tree with squared error.
mediumdocumented fact - Count the minority events in each validation fold before trusting a rare-event metric
An event with about 1% prevalence is validated with 5-fold cross-validation on 3,000 rows, which leaves about 30 events in total and 6 per fold.
mediumpractitioner heuristic - Score anomalies against a peer group, not against the whole population
One detector (isolation forest, local outlier factor, z-score) is fitted on flights, machines or accounts that sit in very different regimes.
mediumpractitioner heuristic - Choose outlier or novelty mode from whether the training data may already contain anomalies
A local outlier factor model is fitted on historical records and then asked to score new records, or its predictions on the training rows are used to evaluate it.
mediumdocumented fact - Evaluate a ranker per list and average, instead of pooling all items into one AUC
A model orders items inside lists (offers for a user, fares for a search, cases for an analyst) and is scored with one AUC or precision across all pooled items.
mediumdocumented fact - Beat a popularity baseline and a repeat-previous baseline before crediting a recommender's personalization
A new recommendation model shows lift in offline hit rate over random or over the current page order.
mediumpractitioner heuristic - Do not present the spread of tree predictions in a forest as a prediction interval
A random forest or bagged model is asked for uncertainty, and the standard deviation or 5th to 95th percentile of its tree predictions is shown as an interval.
mediumpractitioner heuristic - Publish a score contract that says whether the number is a probability, a rank or a margin, and how it was calibrated
Downstream teams consume a model's score in pricing, capacity or campaign logic that they own.
mediumpractitioner heuristic - Recalibrate an average of calibrated models, because averaging probabilities makes the blend underconfident
Models that each pass a reliability check are blended by averaging their probabilities, and the blend feeds an expected-value or threshold rule.
mediumpractitioner heuristic - Summarise a precision-recall curve with average precision, not the trapezoidal area, when scores are coarse
A shallow tree, banded scorecard or isotonic-calibrated model gives only a handful of distinct scores, and its PR-AUC is the trapezoidal area under the curve.
mediumpractitioner heuristic