Feature engineering and preprocessing · insight-1101
Do not treat leave-one-out target encoding as a leakage fix; it still exposes each row's own label
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You replace a category with the mean target of all other rows in that category (leave-one-out) and feed it to a tree model or boosted ensemble.
Action
Use fold-wise cross fitting or ordered statistics instead, and compare the train-versus-holdout gap with the encoded column included and removed.
Reason
Rows of a category share one label total; subtracting the row's own label makes the encoding a monotone function of that label, a leak rather than noise.
Boundary
Repeated categorical levels with learners that can split on tiny differences.
Symptom or risk
Inside a category the encoded value falls as the row's own label rises, so a flexible model can read the label from the encoding; training scores soar and holdout scores do not follow.
Exceptions
Very large categories shrink the per-row shift, so the leak weakens but remains.
Trade-offs
Out-of-fold encodings add variance; ordered schemes need a deliberate row order.
Pitfalls
Adding random noise to the encoding and declaring it safe without measuring the train-versus-holdout gap.
Example
Hypothetical: a route has 1,000 rows and 120 positives. Leave-one-out gives 0.1191 for every positive row and 0.1201 for every negative row, so one threshold separates them perfectly.
Lessons and concepts
Claims and their sources
The CatBoost paper shows that leave-one-out target statistics do not prevent target leakage: for a constant categorical feature a single split on the encoded value classifies the training set perfectly.