Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Adjust predicted probabilities when the class prior shifts but the class-conditional patterns do not
Prevalence in production differs from training, for example no-show rates rising in a holiday season or fraud rates after a new attack wave.
harddocumented fact - Separate prevalence changes from model decay before retraining on a production precision drop
Production precision of an alerting model falls by a third in a month and the team wants to retrain.
hardpractitioner heuristic - Run a fixed leakage playbook when validation looks too good
A first model beats the incumbent by a wide margin, or one feature dominates importance.
hardpractitioner heuristic - Score the full pipeline on shuffled labels to catch leakage the code review missed
A complex pipeline with encoders, selectors and resamplers reports strong performance and you cannot audit every step.
hardpractitioner heuristic - Check LightGBM's monotone constraint method before concluding constraints cost too much accuracy
Adding monotone constraints in LightGBM drops validation performance noticeably and the team wants to remove them.
harddocumented fact - Avoid building reports or monitors on the meaning of individual principal components
A dashboard tracks "PC1 = price sensitivity" over time, or a downstream model consumes PCA outputs that are refitted each month.
hardpractitioner heuristic - Do not treat cross-validation fold scores as independent when judging a small model gap
Two models differ by 0.4 AUC points over ten folds and someone runs a paired t-test on the ten differences.
harddocumented fact - Treat unlabeled cases as unknown, not negative, when positives are only partly recorded
Only investigated cases carry a fraud, defect or violation flag, and every uninvestigated case is coded as a negative.
harddocumented fact - Expect ensemble variance reduction to stall at the correlation between members
A team keeps adding trees, or averaging near-identical models, to reduce variance and sees the gain flatten.
harddocumented fact - Read min_child_weight as a sum of Hessians, not a row count, when moving from squared error to logistic loss
A minchildweight tuned for regression is reused for a binary:logistic model, or is expected to mean at least 20 rows.
harddocumented fact - Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
hardpractitioner heuristic - Validate that ensemble disagreement predicts error before routing cases to review on it
The disagreement among seeds or bootstrap members is proposed as a confidence signal for abstaining or sending cases to manual review.
hardpractitioner heuristic - Correct a share estimated by counting classifier labels when the classifier is imperfect and prevalence moved
A report estimates the share of positive cases, such as cancellations, by counting a classifier's positive labels in a new period.
hardpractitioner heuristic - Weight validation folds by the input density ratio when the scoring population differs in its inputs
You will score a population whose inputs differ from the training mix, such as a new route network, while the input-to-outcome relationship should hold, and cross-validation looks fine.
hardpractitioner heuristic - Compute the prevalence and cost range where a thresholded classifier beats alerting on everyone or no one
A fixed-threshold classifier tuned at one prevalence and cost ratio will be reused where either differs, such as another route group.
hardpractitioner heuristic