LLMs, GenAI, RAG, agents, evaluation · insight-0628
Route counting and aggregation questions to SQL, not to chunk retrieval
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, Data analyst, Data scientist
- Industries
- aviation, retail
Summary: situation, action, reason, boundary
Situation
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
Action
Classify the question type first; send aggregations to a governed query tool over the warehouse (with fixed, reviewed query templates), and reserve RAG for questions answered by passages.
Reason
Top-k retrieval returns a sample of text, not a complete set of rows, so any count computed from it is biased by what happened to rank highly.
Boundary
Assistants that sit over both documents and operational data.
Symptom or risk
Retrieval returns a few reports that mention cancellations, and the model sums whatever numbers appear in them, producing a confident but unfounded total.
Exceptions
When a pre-computed report states the figure explicitly, retrieving and citing that report is fine.
Trade-offs
Free-form text-to-SQL raises safety and correctness risks; start with parameterised templates.
Pitfalls
Letting the model write unrestricted SQL against production tables with write permissions.