LLMs, GenAI, RAG, agents, evaluation · insight-1702
Measure language-ID routing on your own short and mixed messages, and give it a default path
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- telecom, banking
Summary: situation, action, reason, boundary
Situation
An assistant routes each message to a language-specific prompt, analyzer or model after a detector labels the language, and the detector was chosen from its published score.
Action
Label a sample of real traffic by language and by length bucket, measure router accuracy per bucket, set a confidence floor, and send low-confidence or mixed messages to a multilingual default instead of a guess. Log the route taken with each answer.
Reason
Caswell et al. report that language-ID models with high held-out scores gave much lower human-judged accuracy on web-crawled text for many lower-resource languages. Short-message behaviour is not in that abstract, so it must be measured locally.
Boundary
Multilingual support, retrieval with per-language analyzers, and any pipeline that switches prompt or model by language.
Symptom or risk
Short, code-mixed or name-heavy messages are sent down the wrong path, and the failures look like model quality problems on a few customers.
Exceptions
One-language products, or routing by a user-set locale instead of detection.
Trade-offs
The multilingual default is usually weaker than a correct specific path; the confidence floor sets how often it is used.
Pitfalls
Reading one overall accuracy figure, or labelling the test set with the same detector under test.
Example
Hypothetical: the detector is 98% right on the 65% of messages that are long and 80% right on the 35% that are short. Overall accuracy is 0.65 x 0.98 + 0.35 x 0.80 = 91.7%, and about 84% of the 8.3% misroutes come from short messages.
Lessons and concepts
Claims and their sources
Caswell et al. (COLING 2020) report language-ID F1 above 90% in the literature and comparable held-out quality for their models, but only about 5% human-judged accuracy on web-crawl corpora built with them for many lower-resource languages, citing language similarity and domain mismatch among the causes.