Skip to content
ML Academy

Introduced on day 54

LLM cascade

A routing pattern in which a cheaper model answers first and a stronger model is called only when an acceptance check fails.

In detail

Cost is the small model's cost plus the escalation share times the large model's cost, and the tail latency is the escalated path. The saving depends on the acceptance check being reliable and on the escalation share staying stable. FrugalGPT (a 2023 preprint) reports large savings on its authors' tasks; results do not transfer without your own golden set.