LLMs, GenAI, RAG, agents, evaluation · insight-0501
Count tokens per language before sizing prompts, budgets and context limits
- Evidence type
- empirical finding
- Difficulty
- easy
- Roles
- Applied AI, ML engineer
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
Action
Tokenize a sample of real, anonymised messages per language with the deployed model's tokenizer, and set context limits, chunk sizes and budgets from the per-language token counts.
Reason
Subword tokenizers split languages unevenly, so the same content can need several times more tokens in one language than in another.
Boundary
Any multilingual LLM feature with fixed context, latency or per-token budgets.
Symptom or risk
Non-English requests hit context limits, lose retrieved evidence to truncation or cost more per message than planned, and the quality drop is blamed on the model rather than the tokenizer.
Exceptions
Ratios differ by tokenizer; re-measure when you switch models instead of reusing another model's numbers.
Trade-offs
Per-language budgets add configuration; one conservative global budget wastes context for token-efficient languages.
Pitfalls
Estimating tokens from character or word counts calibrated on English text.
Lessons and concepts
Claims and their sources
Petrov et al. found that the same text can require up to about 15 times more tokens in some languages than in others, raising cost and latency and reducing how much fits in context.