LLMs, GenAI, RAG, agents, evaluation · insight-1534
Give the lexical branch a language-specific analyzer for agglutinative or multilingual corpora
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer
- Industries
- aviation, e-commerce
Summary: situation, action, reason, boundary
Situation
A Turkish and English support corpus uses a default English analyzer for BM25, and customers write the same word with different suffixes.
Action
Detect or store document and query language, route each to its language's analyzer (stemming, stop words, casing rules), and evaluate lexical-only recall@k per language slice.
Reason
Lexical matching compares analysed tokens; the analyzer decides whether baggage, baggage's and a Turkish inflected form of bagaj share a term. Elasticsearch ships per-language analyzers, including Turkish.
Boundary
Hybrid retrieval over Turkish, Finnish, German or other morphologically rich or mixed-language text.
Symptom or risk
Inflected forms of one word become different terms, so exact-term matches that hybrid search relies on are missed for the non-English half of the traffic.
Exceptions
Corpora where the lexical branch handles only codes and identifiers, which need exact analyzers rather than stemming.
Trade-offs
Aggressive stemming raises recall and can merge unrelated words, hurting precision.
Pitfalls
Applying one analyzer at index time and another at query time, so tokens never meet.
Lessons and concepts
Claims and their sources
Elasticsearch documents built-in language analyzers aimed at specific languages, including a turkish analyzer, with customisable stop-word and stem-exclusion lists.