LLMs, GenAI, RAG, agents, evaluation · insight-1522
Evaluate instruction language and user language as separate factors before translating the prompt
- Evidence type
- empirical finding
- Difficulty
- medium
- Roles
- Applied AI, Data scientist
- Industries
- aviation, cross-industry
Summary: situation, action, reason, boundary
Situation
A Turkish-language assistant's system prompt is translated into Turkish because the model should be addressed in its users' language.
Action
Run a grid of instruction language by user-message language on natively written test queries, measure task accuracy and answer-language compliance in each cell, and keep one instruction language unless a cell shows a clear gain.
Reason
On the MGSM maths benchmark, Shi et al. found English intermediate reasoning steps gave competitive or better results than steps in the question's language, so the language of instructions and reasoning is a variable to test.
Boundary
Multilingual assistants whose prompts, examples and users use different languages.
Symptom or risk
Quality moves with no way to tell whether instruction language or content language caused it, and every later edit has to be made twice.
Exceptions
MGSM is grade-school maths from 2022-era models and its ten languages do not include Turkish, so no general rule follows; some tasks and newer models behave differently.
Trade-offs
A multilingual test set takes native speakers to build and review.
Pitfalls
Machine-translating English test queries, which are cleaner than the messages real users write.
Lessons and concepts
Claims and their sources
Shi et al. report on MGSM (250 GSM8K problems translated into ten languages) that intermediate reasoning steps in English consistently gave competitive or better results than steps in the native language of the question.