Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Resample inside the cross-validation pipeline so validation folds keep the real class mix
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Plot a learning curve before asking for more data
Stakeholders ask whether another year of history or more labelling budget will improve the model.
easydocumented fact - Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
easydocumented fact - Give early stopping its own validation split and never point it at the test set
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Read the most confident wrong predictions first when debugging a classifier
You start error analysis on a classifier with thousands of misclassifications.
easypractitioner heuristic - Do not score pooled out-of-fold predictions as if they were one test set
You call crossvalpredict, compute one AUC or RMSE over all pooled predictions and report it as the cross-validated score.
easydocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Report permutation importance with its repeat spread and the metric that matches the decision
A permutation importance table shows one number per feature from a single shuffle, scored with the default metric.
easydocumented fact - Compare a model gap with the seed-to-seed spread before declaring a winner
Gradient boosting or forest candidates differ by 0.003 in validation AUC and each was trained once with a single seed.
easyempirical finding - Lower the learning rate and let early stopping choose the tree count
A boosted model with learning rate 0.3 and 200 trees is tuned by hand on several other parameters first.
easydocumented fact - Search scale hyperparameters on a log scale and prefer random sampling to a grid
A grid of C in {0.1, 0.5, 1, 5, 10} and alpha in {0.001, 0.002, 0.003} is searched for a model with six hyperparameters.
easyempirical finding - Verify the sign convention of an anomaly score before ranking, thresholding or comparing detectors
Scores from several detectors (scoresamples, decisionfunction, negativeoutlierfactor) are ranked, averaged or thresholded in one pipeline.
easydocumented fact - Document the label definition and the scored population in the model record, not only the features
A model for churn, no-show or default is handed over, audited or retrained by someone who was not in the original project.
easypractitioner heuristic