Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Publish per-class recall with macro and weighted averages when a multiclass problem has rare classes
A multiclass model (disruption cause, claim type, complaint category) is summarized by accuracy or weighted F1, and one costly class holds a few percent of rows.
easydocumented fact