Neural networks, deep learning, NLP · insight-1450
Evaluate zero-shot cross-lingual transfer on labelled Turkish data before launching in Turkish
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Data scientist, Applied AI
- Industries
- aviation, telecom
Summary: situation, action, reason, boundary
Situation
A multilingual encoder is fine-tuned on English tickets and expected to work on Turkish tickets with no Turkish labels.
Action
Label a few hundred Turkish tickets as an untouched dev and test set, report English and Turkish scores side by side, and compare zero-shot transfer with fine-tuning on a small native set.
Reason
XTREME covers 40 languages and 9 tasks and reports a sizable gap between English and cross-lingually transferred models, largest on syntactic and sentence-retrieval tasks, with wide variation across languages.
Boundary
Any rollout of a model tuned in one language to another.
Symptom or risk
The English dev score says little about Turkish; transfer gaps vary by language and task and are discovered by customers.
Exceptions
Close languages and simple tasks may show small gaps, which you still have to measure.
Trade-offs
Native labels cost money; a small set gives wide intervals.
Pitfalls
Machine-translated test sets that make Turkish look like English; reporting only an average over languages.
Lessons and concepts
Claims and their sources
XTREME (40 languages, 9 tasks) reports a sizable performance gap between English and cross-lingually transferred models, largest on syntactic and sentence retrieval tasks, with wide variation across languages.