Classical ML, validation, metrics, interpretation, debugging · insight-1194
Document the label definition and the scored population in the model record, not only the features
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- telecom, banking, aviation
Summary: situation, action, reason, boundary
Situation
A model for churn, no-show or default is handed over, audited or retrained by someone who was not in the original project.
Action
Record in the model record the exact label rule and window, when the label becomes known, excluded groups with row counts before and after each filter, the eligibility rule at scoring time, and the training date range. Version it with the model and log rows scored outside the eligibility rule.
Reason
The target definition fixes what the model estimates and the eligibility rule fixes where the estimate is valid; features are visible in code, these two are not.
Boundary
Any model that outlives its author or feeds a decision other teams own.
Symptom or risk
The label's window, exclusions and eligibility rule live only in a notebook; a retrain or a new source silently changes what is predicted, and scores get applied to units the model never saw.
Exceptions
Throwaway exploration.
Trade-offs
The record must be updated with every retrain, so it needs a template and a reviewer.
Pitfalls
Documenting the label's name but not its observation window.
Lessons and concepts
Claims and their sources
Model cards are short documents that report a model's performance across relevant conditions and groups, its intended use contexts and its evaluation procedure.