LLMs, GenAI, RAG, agents, evaluation · insight-0672

Route easy requests to a small model and escalate only when a check fails

Situation
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
Action
Build a cascade: a small model answers first, a cheap check (schema validity, retrieval confidence, a verifier score) decides whether to accept, and failures escalate to a larger model; tune the threshold on the golden set.
Reason
FrugalGPT showed that a learned cascade over models of different prices could match the strongest single model at much lower cost on the authors' tasks, because many queries do not need the expensive model.
Boundary
High-volume classification, extraction and FAQ-style traffic.

Symptom or risk

Cost and latency are set by the hardest few percent of requests.

When the acceptance check is unreliable, escalation errors cost more than the savings.

Escalated requests pay for two calls and more latency.

Tuning the router on aggregate accuracy and missing a slice it consistently misroutes.

FrugalGPT's learned LLM cascade matched GPT-4 performance with up to 98% cost reduction on the authors' tasks at 2023 API prices (preprint).
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance