Classical ML, validation, metrics, interpretation, debugging · insight-1188
Treat clicks on a ranked list as position-biased before training or evaluating a ranker on them
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, retail, aviation
Summary: situation, action, reason, boundary
Situation
A ranker is trained on logged clicks or bookings from a page where items were shown in a fixed order.
Action
Log the displayed position, estimate position propensities from a small randomized swap slice, weight clicks by inverse propensity or train with position as a feature fixed at serving, and check offline gains on the randomized slice.
Reason
Observed clicks mix relevance with exposure; without correction the training signal is partly the old ranking.
Boundary
Search, recommendation and offer-ordering systems that learn from their own logs.
Symptom or risk
Higher positions collect more clicks whatever their relevance; the model learns the old ordering, offline metrics reward it for copying the incumbent, and items the old model buried have little data.
Exceptions
Single-item displays or lists where users see everything and position effects are negligible.
Trade-offs
Randomization costs short-term revenue; propensity weights raise variance.
Pitfalls
Estimating propensities from the same biased logs without any randomization.
Lessons and concepts
Claims and their sources
Joachims, Swaminathan and Schnabel describe click feedback as biased by position and propose a counterfactual propensity-weighted ranking approach (a Propensity-Weighted Ranking SVM) to learn from it.