Skip to content
ML Academy

project-044 · LLM and GenAI · cross-industry

Cost and latency routing study: small model, large model, router and cascade on checkable answers

A team runs a high-volume assistant where many requests are easy and some are hard. Would routing easy requests to a cheaper, faster model and escalating the rest keep accuracy close to always using the large model at a lower cost per 1,000 requests? Using a workload with automatically checkable answers, which routing policy sits on the best cost-accuracy frontier under a transparent, hypothetical price table?

Licence
MIT (Hugging Face dataset card)
Reference run
executed

At a glance

  • Data

    GSM8K main config (grade-school math word problems with final numeric answers) (public) · Train 7,473 and test 1,319 problems (dataset card); this project samples 200 + 400

    Details on data

  • Baseline

    Two single-model baselines: always-small and always-large, each run once on the 400 test problems with the same prompt and answer-extraction rule.

    Details on baseline

  • Validation

    Router thresholds and cascade rules are chosen on the 200 train problems only. The 400 test problems are run once per policy;

    Details on validation

  • Metric

    Exact-match accuracy, cost per 1,000 requests (from actual token usage and the hypothetical table), escalation rate, p50/p95 latency and time-to-first-token, all per…

    Details on metric

Success criterion

Pre-set rule: adopt a routing policy only if its accuracy on the frozen test sample is within 2 percentage points of the large model (paired interval) and its cost per 1,000 requests is at least 30% lower under the hypothetical price table, with p95 latency no worse than the large model's. If no policy qualifies, the recommendation is the single model that meets the accuracy floor at the lowest cost.

Context

Day 54 asks for transparent cost/latency trade-offs without assuming bigger is better. GSM8K is a stand-in workload: its answers can be checked exactly, which removes judging noise from the comparison. The conclusions transfer to other workloads only as a method; the measured numbers do not. All prices are hypothetical and labelled; real provider prices change and must be checked at the time of use.

Data

Dataset
GSM8K main config (grade-school math word problems with final numeric answers) (public)
Licence
MIT (Hugging Face dataset card)
Size
Train 7,473 and test 1,319 problems (dataset card); this project samples 200 + 400
Verification
download verified

Access

Hugging Face parquet (openai/gsm8k, main config; no login, not gated). Use a stratified random sample of 400 test problems (fixed seed) for the final comparison and 200 train problems for router development, to control cost.

Grain

One request = one problem; the gold answer is the number after the '####' marker in the answer field.

ColumnMeaning
questionWord problem text sent to the model
answerWorked solution ending with '#### <number>'; only the final number is used for scoring
routeDerived: which model served the request under a policy (small, large, escalated)
tokens_in / tokens_outDerived from the provider's usage fields, not estimated from characters
latency_msDerived: wall-clock time per call, measured client-side, with time-to-first-token if streaming

EDA and data-quality rules

  • Parse the gold number from every answer and assert all parses succeed; normalise commas and units in model outputs with one tested function used for every arm.

  • Profile question length and number of solution steps (lines in the worked answer) as candidate difficulty signals - computed on train only.

  • Contamination note: GSM8K is public since 2021; record each model's stated training cut-off and treat absolute accuracies as possibly inflated.

Baseline

Two single-model baselines: always-small and always-large, each run once on the 400 test problems with the same prompt and answer-extraction rule. Their accuracy, cost and latency define the corners of the frontier.

Features

Router inputs available before the call: question length, count of numbers in the question, presence of rates or percentages, and (optionally) a TF-IDF logistic-regression difficulty classifier trained on train problems labelled by whether the small model solved them. Cascade signal available after the small call: agreement of two small-model samples, or a format/consistency check on the answer. Hypothetical price table: small model 0.20 and large model 3.00 currency units per million input tokens; 0.80 and 15.00 per million output tokens (labelled hypothetical).

Candidate methods

  • Pre-call router (difficulty classifier with a tuned threshold)

    Pays for one call per request; quality depends on predicting difficulty from the question alone.

  • Cascade: small model first, escalate on disagreement between two small samples

    Uses evidence from the cheap model's behaviour; costs extra small calls and adds latency on escalated requests.

Validation design

Router thresholds and cascade rules are chosen on the 200 train problems only. The 400 test problems are run once per policy; small and large model outputs are cached, so policies that reuse the same calls are compared on identical outputs (paired). Bootstrap over problems for accuracy and cost differences. Latency is measured in one session window to limit provider-load effects, and the window is recorded.

Leakage checks

The difficulty classifier's labels come from small-model results on train problems; never on test problems. Thresholds are not adjusted after seeing test results.

Bias and representativeness

GSM8K is English arithmetic reasoning; routing signals that work here (number count) may be meaningless for other workloads. Report this as a method study.

Robustness checks

Sensitivity analysis: recompute the frontier with the price ratio halved and doubled, and with latency at the observed p95 instead of median. A policy that wins only under one price assumption is fragile.

Technical metrics

Exact-match accuracy, cost per 1,000 requests (from actual token usage and the hypothetical table), escalation rate, p50/p95 latency and time-to-first-token, all per policy with paired intervals.

Business interpretation

Cost saved per accuracy point given up, and the share of requests that would see higher latency under a cascade; the decision is presented as a frontier chart, not a single number.

Error slices

  • Difficulty proxies: number of solution steps in the gold answer, question length.

  • Escalated vs non-escalated requests: errors the router let through to the small model.

  • Answer-format failures (unparseable outputs) separately from wrong reasoning.

Implementation steps

  • Load data, parse gold answers, write and test the answer-extraction function.

  • Build the cost calculator from usage fields and the labelled hypothetical price table; unit-test it.

  • With configured small and large models: run both on the 200 train problems, build the difficulty labels and router; cache all outputs with model ids and timestamps.

  • Run single models, router and cascade on the 400 test problems; compute the frontier and sensitivity analysis.

  • Write the recommendation, including what would change it (price changes, a new small model).

Reproduction

Learner-authored; for example uv run python -m routing.prepare --seed 11 then uv run python -m routing.run --policy all --cache runs/ with model ids set in environment variables. Without models, only the data preparation, cost calculator and a simulation of the frontier under assumed accuracies run; such a simulation must be labelled "simulated with assumed accuracies" and cannot support a routing decision.

Limitations

  • Hypothetical prices: the frontier's shape depends on the price ratio, which differs by provider and changes over time.

  • Benchmark contamination may inflate both models' accuracy and shrink the measured gap.

  • Latency measured from one location in one time window does not represent provider load variation.

Production and monitoring

Log the route decision, model ids, token usage and latency per request; monitor escalation rate and accuracy on a sampled, labelled stream. Re-tune thresholds when either model changes. Keep a kill-switch that sends all traffic to the large model if quality alarms fire.

Interview questions

  • How would you set up a routing experiment so that cost differences are not confounded by different prompts or answer parsing?

  • When does a cascade cost more than always using the large model?

  • Your router saves 40% under today's prices. What happens to the recommendation if the small model's price doubles?

Postmortem and alternatives

If the router barely beats random routing at the same escalation rate, difficulty is not predictable from the question and the cascade (or a single cheaper model) is the honest answer. Discuss caching and prompt shortening as cost levers that do not risk quality.

Reference implementation

Status: executed — outputs below are from the recorded run

cd python && uv run python -m reference.r20_cost_latency_routing.run

r20 results - cost and latency routing study on a synthetic world

Synthetic world, hypothetical prices, assumed latencies: this is not a measurement of any real model, price or provider. Per-request outcomes are drawn from an authored quality-by-difficulty table (world.py, seed 20260044) and stored in fixtures/recorded_world_v1.json. The project record states that a simulation with assumed accuracies "cannot support a routing decision": the recommendation below is the output of the pre-set rule on this synthetic world, shown to demonstrate the procedure. Numbers are copied from outputs/metrics.json and the CSV files written by cd python && uv run python -m reference.r20_cost_latency_routing.run.

ItemValue
Date, seed2026-10-12; world seed 20260044 (replicate worlds 20260045 to 20260244); bootstrap seeds fixed
Requests200 dev + 400 held-out (headline world); 200 further worlds of the same size for the replicate study
PricesHYPOTHETICAL: small 0.20 in / 0.80 out, large 3.00 in / 15.00 out per million tokens
EnvironmentPython 3.12.13, numpy 2.5.3, matplotlib for the chart
Runtime16 s full run (about 2 s --smoke); tests 5 s

Router: logistic regression on length, number count and a rate cue, trained on the 200 dev requests. AUC for "small model is right": 0.778 on dev (in-sample), 0.688 held-out; the true number of solution steps used as a score would give 0.718. In the authored world a request's difficulty is only loosely visible before the call.

Headline world: held-out 400 requests

Thresholds were chosen on the 200 dev requests (cheapest grid point within 2 points of the large model's dev accuracy of 0.905). Intervals are paired bootstrap over requests; accuracy bound is one-sided 95%, others two-sided 95%.

PolicyAccuracy (diff vs large)Lower 95% bound of diffCost per 1,000 (saving vs large)p50 / p95 latency (s)EscalationRule: accuracy / cost / p95
always_small0.753 (-0.163)-0.2030.17 (96.0%)1.58 / 2.810%fail / pass / pass
always_large0.9154.265.93 / 12.00100%pass / fail / pass
router, tau 0.81 (dev-selected)0.873 (-0.043)-0.0632.87 (32.5%, 28.5 to 36.7)5.24 / 11.33 (-0.66)58%fail / pass / pass
router+cascade, 0.40 / 0.80 (dev-selected)0.873 (-0.043)-0.0681.79 (58.0%, 53.0 to 62.9)1.82 / 11.34 (-0.66)30%fail / pass / pass
cascade (two small samples agree)0.925 (+0.010)-0.0082.02 (52.5%, 47.5 to 57.3)2.16 / 12.39 (+0.39; -3.74 to +2.01)36%pass / pass / fail
oracle router (not deployable)0.963 (+0.048)+0.0301.31 (69.2%)1.73 / 9.2725%pass / pass / pass

Decision under the pre-set rule: no routing policy qualifies, so the recommendation for this synthetic world is the single model that meets the accuracy floor at the lowest cost: always_large.

Findings

  1. The dev-selected thresholds did not survive held-out evaluation. Both routers met the dev floor (dev accuracy 0.895 and 0.885 against 0.905 for large, i.e. within 2 points by selection) and lost 4.3 points on the held-out requests (lower bound -6.3 and -6.8). Selecting the cheapest point that just meets a floor on 200 requests picks points that are optimistic on dev (a winner's curse); this is why the floor is not enough on dev.

  2. The cascade is accurate and cheap but fails the p95 rule. Accuracy +1.0 point (one-sided bound -0.8, inside the -2 point margin), 52.5% cheaper, p50 latency 2.16 s against 5.93 s, but p95 is 0.39 s (3.2%) worse than the large model because the 36% of requests that escalate pay a small call plus a large call; the interval for the p95 difference (-3.7 to +2.0 s) includes zero. Whether "p95 no worse than large" should be a point rule is a judgment: with a 5% tolerance the cascade would qualify in this world (post-hoc, not the pre-set rule).

  3. A cascade costs more than the large model only at very high escalation here. With the average token costs of this workload and the hypothetical table, two small samples plus escalation exceed the large model's cost only when more than 92% of requests escalate (84% if the large model's price is halved; 60% if the small price is five times higher). The practical risk is latency, not cost.

  4. Router versus chance. The router that minimises dev total cost at 0.20 per wrong answer (tau 0.90) escalates 80% of requests and reaches accuracy 0.900; random routing at the same escalation rate gives 0.884 (+1.6 points for the router). The cascade (accuracy 0.925 at 36% escalation) is far above random routing at its rate (0.811): its signal is informative but costs two small calls per request. The oracle router (not deployable) saves 69% at accuracy 0.963; the tau 0.90 router saves 14% (3.65 against 4.26 per 1,000 requests) at accuracy 0.900.

  5. Routing error costs are asymmetric in counts, not necessarily in money. For the tau 0.90 router on 400 held-out requests: 320 sent to large, 80 to small. 8 missed escalations (small wrong, large would have been right; 4.0 per 1,000 requests at the hypothetical 0.20 per wrong answer), 1 unavoidable (both wrong), and 230 unnecessary escalations (small was right; 2.41 extra compute per 1,000 requests). The error rate among requests routed to small is 11.3% against 28.1% among those routed to large, so the router does separate difficulty, weakly.

  6. The preferred policy depends on what a wrong answer costs. Total cost per 1,000 requests (compute plus errors, thresholds chosen on dev, evaluated held-out):

    Cost per wrong answer (hypothetical)always_smallalways_largecascaderouterrouter+cascadeCheapest
    0.025.125.963.525.19 (tau 0.74)3.60cascade
    0.2049.6721.2617.0223.65 (tau 0.90)16.20router+cascade
    2.00495.17174.26152.02203.65 (tau 0.90)142.20router+cascade

    The small model alone is never cheapest once a wrong answer costs 0.02 or more; the router alone is never cheapest here; its dev-chosen points are dominated by the cascade.

  7. Price assumptions move the savings, not the recommendation, in this world. Cost saving versus large for router / router+cascade / cascade: base 32% / 58% / 52%; large price x0.5 (same as small price x2, equal ratio) 31% / 53% / 45%; large price x2 33% / 61% / 57%; small price x5 27% / 37% / 21%. The recommendation stays always_large in all five scenarios because the accuracy and p95 conditions fail, not the cost condition; at small price x5 the cascade would also fail the 30% cost condition.

  8. Cache hit rate shrinks absolute savings and raises the break-even volume. Exact-repeat cache with hit rate h (hits cost nothing and take 0.02 s; HYPOTHETICAL fixed costs 300 per month for the cascade, 3,300 for router+cascade):

    Hit ratelarge cost/1,000cascade cost/1,000cascade saving/1,000cascade break-even requests/monthrouter+cascade break-even
    0%4.262.022.24134,0001.60 M
    10%3.851.782.07145,0001.73 M
    20%3.451.591.86161,0001.95 M
    40%2.561.091.47204,0002.44 M
    60%1.610.670.94321,0003.94 M

    Always-large with a 60% hit rate (1.61) is cheaper than the uncached cascade (2.02) with no quality risk, if the workload really has that many exact repeats. p95 latency falls with the hit rate (12.0 s to 8.7 s for large at 60%).

  9. By difficulty. Accuracy (small / large / cascade / tau 0.90 router) is 0.916 / 0.985 / 1.000 / 0.962 on the 131 requests with 2 to 3 steps, 0.755 / 0.903 / 0.942 / 0.884 on the 155 with 4 to 5 steps and 0.561 / 0.851 / 0.816 / 0.851 on the 114 with 6 to 8 steps. The cascade escalates 15%, 36% and 60% of these groups; it is below the large model only on the hardest group (-3.5 points) where two wrong small samples that happen to agree pass as consistent.

  10. Replicate worlds (200 independent worlds with the same assumptions).

    FamilySelected on devQualifies on held-outFails accuracyFails costFails p95Median accuracy diff (p10 to p90)Median savingMedian p95 diff (s)
    router20001611388-2.3 (-5.3 to -0.5) points24.8%-0.87
    router+cascade200151680110-2.3 (-6.3 to +0.3) points55.2%+0.15
    cascade17925910127-0.3 (-2.0 to +1.3) points51.7%+0.63

    The rule recommends always_large in 168 of 200 worlds (84%), the cascade in 19 and router+cascade in 13. The headline world is therefore a typical, not an unlucky, draw. Post-hoc variants (not the pre-set rule): a 5% p95 tolerance lets router+cascade qualify in 20 worlds and the cascade in 39; requiring the dev accuracy to be at least the large model's (margin 0 instead of 2 points) gives always_large 162, cascade 8, router+cascade 30. The accuracy bound is structural: selecting at the floor on dev and then requiring a paired lower bound above the floor on 400 requests passes only when the true difference is near zero or positive.

  11. The frontier chart (outputs/frontier.png) is optimistic as drawn. 226 grid policies are plotted on the same 400 held-out requests; 98 are non-dominated in accuracy, cost and p95 together (30 in accuracy and cost), mostly because p95 from 400 requests is noisy. Picking any of those points would repeat the selection-on-test error that the dev/held-out split exists to prevent.

Pre-set rubric items and what is unmet

ItemStatus
Cost from real token usage and a labelled hypothetical price tablePartly: token counts and prices are authored; no provider usage fields exist. Prices are labelled.
Latency measured, not assumedNot met: latencies are simulated from an assumed model. Cannot be met offline.
Same prompt and parser across arms, cached outputs, paired intervalsMet by construction (every policy reuses the same recorded calls; paired bootstrap).
Router compared with random routing at the same escalation rateMet (finding 4).
Frontier chart and a policy chosen by the pre-set ruleMet on the synthetic world; the choice (always_large) is not a statement about real models.
Sensitivity to price ratiosMet (finding 7); latency at p95 alongside median is reported for every policy.
Re-evaluation triggers and guardrailsIn the project record's production note; not exercised here.

What this does not prove

  • It says nothing about any real model, provider, price or latency. Every accuracy, token count and latency comes from an authored table; change the table and the recommendation changes. Prices are hypothetical and real prices must be checked at the time of use.
  • The replicate worlds vary sampling noise only. They cannot show how the conclusion moves with a different quality-cost curve, a richer routing signal or heavier latency tails.
  • The "no policy qualifies" outcome follows from three authored properties together: a weak pre-call signal (AUC 0.69), a large model that is better at every difficulty, and a strict p95 rule. A workload with an informative difficulty signal would route better.
  • The router and thresholds were fitted on 200 dev requests. The held-out evaluation used 400; p95 and intervals from that few requests are wide and the frontier as drawn is selected on the same requests it is evaluated on.
  • Cost-per-wrong-answer, fixed monthly cost and cache hit rates are hypothetical illustrations of the arithmetic.
  • Format failures, retries, rate limits, batching and provider load are not modelled.

Reproduce

cd python && uv run python -m reference.r20_cost_latency_routing.run
cd python && uv run pytest tests/test_r20_cost_latency_routing.py

Builder-authored reference. It is not learner evidence and is never credited to your portfolio.

Rubric3 criteria

CriterionMeetsExceeds
Transparent economicsCost computed from real token usage and a labelled hypothetical price table; latency measured, not assumed.Sensitivity analysis shows how the decision changes with price ratios.
Fair comparisonSame prompt and parser across arms; cached outputs; paired intervals.Router compared with random routing at the same escalation rate.
RecommendationFrontier chart and a policy chosen by the pre-set rule.States re-evaluation triggers and production guardrails.

References5