Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Resample inside the cross-validation pipeline so validation folds keep the real class mix
You undersample or oversample the whole dataset, then cross-validate or split into train and test.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Plot a learning curve before asking for more data
Stakeholders ask whether another year of history or more labelling budget will improve the model.
easydocumented fact - Read a validation curve for one hyperparameter to locate under- and overfitting, then confirm on held-out data
You are unsure whether to increase or decrease tree depth, regularization strength or the number of neighbours.
easydocumented fact - Give early stopping its own validation split and never point it at the test set
You train LightGBM, XGBoost or a neural network with early stopping and pass the test set as the evaluation set.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Read the most confident wrong predictions first when debugging a classifier
You start error analysis on a classifier with thousands of misclassifications.
easypractitioner heuristic - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Do not score pooled out-of-fold predictions as if they were one test set
You call crossvalpredict, compute one AUC or RMSE over all pooled predictions and report it as the cross-validated score.
easydocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Report permutation importance with its repeat spread and the metric that matches the decision
A permutation importance table shows one number per feature from a single shuffle, scored with the default metric.
easydocumented fact - Compare a model gap with the seed-to-seed spread before declaring a winner
Gradient boosting or forest candidates differ by 0.003 in validation AUC and each was trained once with a single seed.
easyempirical finding - Lower the learning rate and let early stopping choose the tree count
A boosted model with learning rate 0.3 and 200 trees is tuned by hand on several other parameters first.
easydocumented fact - Search scale hyperparameters on a log scale and prefer random sampling to a grid
A grid of C in {0.1, 0.5, 1, 5, 10} and alpha in {0.001, 0.002, 0.003} is searched for a model with six hyperparameters.
easyempirical finding - Publish per-class recall with macro and weighted averages when a multiclass problem has rare classes
A multiclass model (disruption cause, claim type, complaint category) is summarized by accuracy or weighted F1, and one costly class holds a few percent of rows.
easydocumented fact - Verify the sign convention of an anomaly score before ranking, thresholding or comparing detectors
Scores from several detectors (scoresamples, decisionfunction, negativeoutlierfactor) are ranked, averaged or thresholded in one pipeline.
easydocumented fact - Document the label definition and the scored population in the model record, not only the features
A model for churn, no-show or default is handed over, audited or retrained by someone who was not in the original project.
easypractitioner heuristic