Classical ML, validation, metrics, interpretation, debugging · insight-1187
Beat a popularity baseline and a repeat-previous baseline before crediting a recommender's personalization
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, retail, aviation
Summary: situation, action, reason, boundary
Situation
A new recommendation model shows lift in offline hit rate over random or over the current page order.
Action
Build two baselines on the same split: top-N popularity in the latest window and, where repeats make sense, each user's most recent or most frequent items. Report lift over both, split by user history length (new, light, heavy).
Reason
The personalized signal is what remains after popularity and repetition are accounted for.
Boundary
Product, route, content and offer recommendation with logged interactions.
Symptom or risk
Much of the gain can come from head items or from repeating what the user already chose; the personalized model may add little beyond them while carrying its full cost.
Exceptions
Catalogues where repeats are undesirable, such as a fresh-content feed; drop the repeat baseline there.
Trade-offs
Strong baselines make most models look weak, which is the point but can be unwelcome.
Pitfalls
Computing popularity on the whole period including the test window.