Neural networks, deep learning, NLP · insight-1458
Exclude text written after the outcome, such as resolution notes and closing remarks
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
You predict at ticket creation whether a complaint will be escalated, using a text column that holds the full conversation including later agent notes.
Action
List when each text field is written, rebuild the input as of the prediction moment from timestamped events, and inspect the top-weighted n-grams for outcome words.
Reason
Kaufman et al. define leakage as using information in training that will not be legitimately available when the model is applied; text fields are a common carrier.
Boundary
Tickets, claims, case notes and clinical or CRM free text.
Symptom or risk
Phrases such as "forwarded to supervisor" or "refund issued" exist only after the outcome, so offline scores are near perfect and the model is useless at creation time.
Exceptions
Models run after resolution, such as quality review, may use all text.
Trade-offs
Timestamped event history is harder to assemble than a final text column.
Pitfalls
Trusting a field's name; counting an edited or appended field as available at creation.
Example
Hypothetical: dropping agent notes written after the first reply moves a model from 0.97 to 0.71 AUC, and 0.71 is the number production will see.
Lessons and concepts
Claims and their sources
Kaufman et al. formulate leakage as the introduction of information about the target that would not legitimately be available at prediction time.