LLMs, GenAI, RAG, agents, evaluation · insight-0626
Decompose multi-hop questions instead of hoping one retrieval finds everything
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Applied AI, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
Action
Detect multi-part questions, issue one retrieval per sub-question (generated or templated), and assemble the evidence with clear labels; measure "all required evidence in context" rather than any-hit recall.
Reason
One query embeds one information need. ReAct interleaves reasoning with retrieval actions on multi-hop QA so missing facts are fetched step by step, and its authors report fewer hallucinations than reasoning alone.
Boundary
Comparison, join-like and multi-entity questions.
Symptom or risk
A single query retrieves the rule or the fleet list but rarely both, so the model fills the gap with a guess.
Exceptions
When the second hop is structured data (fleet lists, schedules), a database query is more reliable than a second text retrieval.
Trade-offs
Each extra hop adds latency, tokens and another place to fail; cap the number of hops.
Pitfalls
Evaluating multi-hop questions with single-evidence recall, which overstates success.
Lessons and concepts
Claims and their sources
ReAct interleaves reasoning traces with actions such as Wikipedia API calls on multi-hop QA (HotpotQA) and fact verification (FEVER), and its authors report that this reduces the hallucination and error propagation seen in chain-of-thought reasoning.