LLMs, GenAI, RAG, agents, evaluation · insight-1725
Score generated SQL on several database variants, not on the one database you have
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Applied AI, ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A text-to-SQL feature is scored by running the generated and reference queries on one database and comparing results.
Action
Per schema, build a few databases with the same tables, keys and constraints but different values, ties and edge cases. Count a query correct only if it matches the reference on all of them.
Reason
Zhong, Yu and Klein score result sets on a small distilled suite of databases, a tight upper bound on semantic accuracy; one database is a looser bound.
Boundary
Evaluation sets and tests for generated SQL.
Symptom or risk
A wrong query that matches the reference on that data counts as correct; accuracy is overstated and errors reach production.
Exceptions
Answers defined only on live data, or a schema too small to vary.
Trade-offs
Variants must respect constraints (subsamples are a crude stand-in); more of them cost runtime and do not prove equivalence.
Pitfalls
Values that break foreign keys; variants so alike they act as one.
Example
Course run on the bundled airline sample (not a benchmark). Reference: SUM(revenue) by route; wrong query: AVG(avg_fare). Both return IST-JFK on the full table; on 200 random 50-row subsamples they disagreed in 34 to 49% of draws by seed, so ten independent variants catch it about 99% of the time.
Lessons and concepts
Claims and their sources
Zhong, Yu and Klein propose test suite accuracy for text-to-SQL, the denotation accuracy of predicted queries on a small suite of databases distilled for coverage of the gold query from many randomly generated ones, as an approximation of semantic accuracy; the abstract calls it a tight upper bound.