Neural networks, deep learning, NLP · insight-1457
Split text data by thread or customer so quoted replies and personal style cannot cross the boundary
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, saas
Summary: situation, action, reason, boundary
Situation
Support emails are split by message at random, though each thread has several messages that quote earlier ones and come from the same customer.
Action
Build a group key from thread id (customer id as fallback), use group-aware splitting, strip quoted blocks, and report the gap between random and grouped scores.
Reason
Elangovan et al. show that train-test overlap inflates NLP results by rewarding memorisation, and scikit-learn documents group-aware iterators that never place one group in both training and test.
Boundary
Email, chat, review and ticket text with repeated authors or conversations.
Symptom or risk
Quoted blocks and shared phrasing put near-copies of training messages into the test split, so the score measures memorisation of a thread and a customer's habits.
Exceptions
If production messages always come from customers already in training, a random split is closer to deployment.
Trade-offs
Fewer distinct groups per fold raises variance; large threads make folds uneven.
Pitfalls
Grouping by message id; deduplicating only exact copies while quotes differ by a few tokens.
Lessons and concepts
Claims and their sources
Elangovan et al. quantify how train-test overlap in NLP datasets inflates results by rewarding memorisation.
scikit-learn documents GroupKFold as guaranteeing the same group is never in both the training and testing set of a split.