Classical ML, validation, metrics, interpretation, debugging · insight-0161
Split by entity with grouped cross-validation when the same customer, aircraft or patient appears many times
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- healthcare, aviation, retail
Summary: situation, action, reason, boundary
Situation
Rows are repeated measurements of the same entity, and the model will score new entities in production.
Action
Use GroupKFold or StratifiedGroupKFold with the entity id as the group, and compare the grouped score with the random-split score to size the optimism.
Reason
Validation must mimic deployment: if production entities are unseen, validation entities must be unseen too.
Boundary
Customers with many orders, flights recurring daily, machines with many sensor windows, patients with repeat visits.
Symptom or risk
Random K-fold puts the same entity in training and validation; the model recognizes entities and scores look far better than for newcomers.
Exceptions
If production scores the same known entities over time, an out-of-time split with entity overlap can be the right design.
Trade-offs
Grouped folds have uneven sizes and noisier estimates, especially with a few dominant groups.
Pitfalls
Grouping by a too-fine key (booking id) when correlation runs at a coarser level (customer or household).
Lessons and concepts
Claims and their sources
GroupKFold keeps each group in either the training or the test side so the score reflects generalization to unseen groups; StratifiedGroupKFold also approximates class balance.
Non-independence between training and test samples is a recognized source of leakage in ML-based science.