Neural networks, deep learning, NLP · insight-0464
Watch for vocabulary drift in embedding-based models and adapt the encoder only when it pays
- Evidence type
- empirical finding
- Difficulty
- hard
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A complaint-topic clustering or classification model built on general-purpose sentence embeddings degrades after a new loyalty programme and new fare products introduce unfamiliar jargon.
Action
Track, per week, the share of unseen domain terms and the nearest-neighbour similarity of new texts to the training set. When they shift, label a fresh sample, measure the drop, and compare a lexical baseline with in-domain adaptation of the encoder (continued pretraining or fine-tuning on domain pairs) before retraining downstream models.
Reason
Dense encoders often lose quality out of domain, and continued in-domain pretraining has improved downstream classification across several domains.
Boundary
Embedding-based classifiers, clustering and search over evolving specialist text.
Symptom or risk
Unfamiliar domain terms land in poorly separated regions of the embedding space, so new complaints are mixed into wrong topics while dashboards still look stable.
Exceptions
Stable vocabularies with little new terminology rarely need encoder adaptation.
Trade-offs
Adaptation costs compute and creates a model you must maintain and re-embed with.
Pitfalls
Adapting the encoder but keeping the old downstream classifier trained on the previous vectors.
Lessons and concepts
Claims and their sources
Thakur et al. report that dense retrieval models often underperform out of domain in zero-shot evaluation, while BM25 is a robust baseline.
Gururangan et al. show that a second phase of in-domain pretraining improves performance across four domains and eight classification tasks.