Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Score a dummy baseline on the same split and metric before reading any model result
A first model reports 0.93 accuracy or an MAE of 4.1 and the team is ready to celebrate.
easydocumented fact - Benchmark against the incumbent business rule, not only against a trivial baseline
A model beats a dummy baseline and you are drafting the launch proposal.
easypractitioner heuristic - Replace accuracy with prevalence-aware metrics when the positive class is rare
You report accuracy for fraud, churn, defect or no-show prediction where positives are a few percent.
easydocumented fact - Read PR-AUC against its prevalence baseline and never compare it across populations with different base rates
You compare PR-AUC of a churn model across countries, months or customer tiers.
mediumdocumented fact - Do not compare R² across test sets with different target variance
A model's R² is 0.85 on the network-wide test set and 0.40 on one route, or differs between two quarters.
mediumdocumented fact - Retire MAPE when actuals can be zero or near zero
You evaluate demand, sales or bookings where some actuals are zero or tiny, for example small routes or slow SKUs.
easydocumented fact - Expect MAPE-selected models to under-forecast and check bias before adopting them
You choose between models or tune a model by minimizing MAPE on positive targets.
mediumpractitioner heuristic - Pick MAE or RMSE from whether the decision needs the median or the mean
You are choosing a training loss or headline metric for a regression model.
easydocumented fact - Derive the decision threshold from the cost matrix instead of using 0.5
A calibrated classifier will trigger an action (block a transaction, call a customer, deny an overbooking) and someone asks what cut-off to use.
mediumdocumented fact - When reviewers can handle only K cases, rank and evaluate precision at K instead of choosing a probability cut-off
An operations team can investigate a fixed number of alerts per day regardless of how many the model flags.
mediumpractitioner heuristic - Deduplicate at the correct grain before splitting so copies do not straddle train and test
The raw table has exact or near-duplicate rows, for example re-sent records or late updates of the same event.
easydocumented fact - Remember scikit-learn's logistic regression is penalized by default before comparing it with statistics output
Coefficients from scikit-learn's LogisticRegression differ from statsmodels or R on the same data, or are used for inference.
easydocumented fact - Do not read lasso's choice among correlated features as evidence that the others do not matter
Lasso keeps one of several correlated features (competitor price, fare level, season) and zeroes the rest; someone proposes dropping them from the data.
mediumdocumented fact - Interpret predictions, not individual coefficients, when features are strongly collinear
Coefficients change sign or size when a correlated feature is added, removed or the data period shifts.
mediumdocumented fact - Treat a funnel-shaped residual plot as a modelling decision, not cosmetic noise
Residuals versus fitted values widen as predictions grow, common for revenue, demand and claim size.
mediumdocumented fact - Plot residuals in time order even for models that are not forecasts
A cross-sectional regression is trained on data collected over months, such as flight-level revenue or store-level sales.
mediumdocumented fact - Rank error slices by total cost, not by error rate alone
An error-analysis table lists dozens of segments with their error rates and the team must choose what to fix first.
easypractitioner heuristic - Refuse to turn SHAP values into intervention advice without a causal design
A stakeholder sees that a feature has large SHAP values and asks you to change that feature to improve the outcome.
mediumdocumented fact - Establish that the model predicts well before discussing which features it relies on
An importance plot is requested for a model whose held-out score is close to the baseline.
easydocumented fact - Plot ICE curves with the PDP so averages do not hide opposite effects
A partial dependence plot shows a flat or mild effect for a feature that domain experts expect to matter.
mediumdocumented fact - Test whether clusters survive resampling before naming them
A k-means run on customer or route features produces k tidy segments that the business wants to name and target.
mediumdocumented fact - Validate segments on outcomes they were not built from before calling them personas
A clustering of customers or routes is about to become a set of named personas with tailored strategies.
mediumpractitioner heuristic - Derive the false alarm rate an anomaly detector must reach from the base rate and the precision you need
An anomaly detector shows 95% detection with a 1% false alarm rate on a test set, and anomalies occur in 0.1% of cases in production.
mediumdocumented fact - Size recall and precision intervals by the number of positives, not by the number of rows
A test set has 50,000 rows and 40 positives, and the report quotes recall of 0.80 with no interval.
easypractitioner heuristic - Let the stake of each case set its own action threshold when costs scale with value
A single probability cut-off is used for retention offers, refunds or fraud holds although the money at risk varies by a factor of fifty between cases.
mediumpractitioner heuristic - Pick a threshold in the flat part of the expected-cost curve, not at its sharpest minimum
Your cost curve over thresholds has a minimum at 0.19 but the unit costs behind it are rough estimates.
mediumpractitioner heuristic - Measure how often two labelers agree before treating a score gap as progress
Labels come from analyst judgement or agent coding and the team tunes models for another point of accuracy.
mediumpractitioner heuristic - Treat huge logistic coefficients and unstable standard errors as a sign of separation
A logistic regression returns coefficients of 15 or 30 for a dummy variable, enormous standard errors or a convergence warning.
mediumdocumented fact - Choose the simplest regularization setting within one standard error of the best
Cross-validation picks the lowest-error alpha from a path of 100 values and the error curve is flat across a wide stretch.
mediumpractitioner heuristic - Plot the coefficient path before naming the drivers a lasso model selected
A lasso fit at one alpha keeps eight features and the report calls them the key drivers.
mediumdocumented fact - Judge a slice's error rate against sample-size control limits before calling it the problem
A slice table sorted by error rate puts a 20-row segment at the top with 25% errors while the model averages 10%.
mediumdocumented fact - Judge an interaction on the decision's scale: no log-odds interaction can still mean unequal probability gaps
A logistic model has two drivers, such as booking window and market, and the team asks whether they interact using the p-value of the product term.
hardpractitioner heuristic - Compare Pearson chi-square to residual degrees of freedom before trusting Poisson standard errors
A Poisson regression for counts (no-shows per flight, tickets per shift) returns narrow intervals and very significant drivers.
mediumdocumented fact - Choose the number of clusters from the action you can take and a stability check, not from inertia
A segmentation run reports inertia for k from 2 to 15 and the team names segments at the apparent elbow.
mediumdocumented fact - Handle DBSCAN's noise label as an explicit population in downstream tables, not as a cluster or a discard
DBSCAN labels are joined back to a customer or sensor table and label -1 is used in group-bys, one-hot encoded as cluster -1, or dropped.
mediumdocumented fact - Cluster in the original feature space, not on a t-SNE map, and use the map only to look
A t-SNE picture of customer or booking features shows islands, and k-means is run on its two coordinates or segment sizes are read off the plot.
mediumempirical finding - Publish per-class recall with macro and weighted averages when a multiclass problem has rare classes
A multiclass model (disruption cause, claim type, complaint category) is summarized by accuracy or weighted F1, and one costly class holds a few percent of rows.
easydocumented fact - Model an ordered target with ordered thresholds and score it by how many levels it misses
The target has ordered levels, such as complaint severity 1 to 5, and you are about to fit an unordered multiclass model and report accuracy.
mediumpractitioner heuristic - Fit a logit-link model to a bounded share such as load factor instead of a straight line
The target is a share between 0 and 1, such as load factor, and a linear regression predicts values below 0 or above 1.
mediumpractitioner heuristic