Classical ML, validation, metrics, interpretation, debugging · insight-1138
Build learning curves by number of entities, not number of rows, when rows cluster
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- retail, saas, aviation
Summary: situation, action, reason, boundary
Situation
Each customer, aircraft or flight contributes dozens of rows and the learning curve is built by subsampling rows.
Action
Subsample whole entities for the training sizes (learning_curve itself subsamples rows, so build the entity-level sizes yourself), score with a group-aware splitter on entity ids, and plot score against entities.
Reason
Deployment scores unseen entities, so what limits performance is how many different entities the model has seen, and rows of one entity are strongly correlated.
Boundary
Panel and repeated-measures data: customers, devices, routes, patients.
Symptom or risk
Extra rows from entities already seen add little new information, so the curve looks flat and suggests more data is useless, or looks steep because near-copies leak across the split.
Exceptions
If every new row is a new entity (one row per customer), rows and entities coincide.
Trade-offs
Entity-level curves are noisier with few entities; repeat with several random subsets.
Pitfalls
Using group-aware folds for scoring but a row-level size grid, which varies two things at once.
Example
Hypothetical: 500 customers with 20 bookings each; going from 5,000 to 10,000 rows by adding bookings of the same customers adds no new customers.
Lessons and concepts
Claims and their sources
learning_curve takes group labels (groups) that are used when splitting and only with a Group cross-validation instance such as GroupKFold; GroupKFold keeps the same group out of both training and testing sets.