S15 · rag eval simulation

Turkish questions against English policy pages

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

A synthetic airline's assistant answers agent questions from 2,100 English policy pages. The English pages are the authoritative text; every answer must cite the English passage. Support logs (simulated) show 65% of queries in English and 35% in Turkish.

Constraints (hypothetical): p95 latency at most 3.0 seconds, cost at most 2.00 USD per 1,000 queries, and no more than a 2-point drop in English retrieval recall. All results below are synthetic.

Stage 1 of 4

Agents say: "Turkish questions about change fees often come back as not covered." The current system embeds queries and passages with an English-centric model and returns the top 5 passages. A simulated engineer proposes translating the whole corpus into Turkish immediately.

What is your first move?

Translate the corpus into Turkish and index both languages, since a model cannot match text across languages it was not trained to align, and the translation can be produced in a single batch run.Add a line to the system prompt telling the model to answer Turkish questions from the English passages it is given, since it reads both languages well enough to bridge them itself.Build a labelled set stratified by query language (100 English, 50 Turkish from real log themes), report recall per language and at the 65/35 production mix, then compare fixes on the slice agents complain about.Swap in the largest embedding model available, since larger models cover more languages and need no further evaluation, which also avoids the cost of building a Turkish test set.