Introduced on day 41
Near-duplicate leakage
Inflated evaluation scores caused by almost identical messages (resubmissions, forwards, templates) appearing on both sides of a train and test split.
In detail
The model is scored on text it has effectively memorised, so offline accuracy overstates production accuracy. Guard against it by splitting on a group key such as ticket, customer or template, by measuring the nearest-neighbour similarity between test and training messages, and by deduplicating near-copies when no group key exists.