Core day 54 of 60 · Week 11 · Part II · application depth

AI Cost, Latency and Reliability

Finance says the assistant costs too much, the product owner says it answers too slowly, and an engineer proposes the largest model "to be safe". What do you compute before changing anything, and why is "which model is best" the wrong question?

By the end: From token counts, editable (hypothetical) prices and speeds, and a simulated pass rate, you compute cost per 1,000 requests and per good answer, split latency into first token and generation, and pick a configuration from the quality-cost frontier under stated constraints.

At a glance

Price the good answer, split latency into first token and generation, and pick the cheapest configuration that clears every gate.

Two token meters
Input drives first-token wait and RAG cost; output drives generation time and costs more per token.
Per good answer
Divide by the pass rate and count retries and escalations: cost per successful task.
Split the wait
TTFT grows with input; generation grows with output; streaming changes only the first.
Levers
Caching, batch, shorter context, capped output, routing: each moves a different meter.
Watch the tail
A cascade lowers mean cost and latency while its p95 becomes the escalated path.

Recall · 5 min

Without notes: what does a retried LLM call cost you beyond the first attempt (Day 52), and why does a quality number on a golden set need slice counts before you trust it (Day 51)?

Open due reviews

Concept · 10 min

Two meters run on every request: one turns tokens into money, the other turns them into waiting. Input tokens drive the wait for the first word, output tokens the wait for the last, and output usually costs several times more per token. Quality never shows on an invoice. So you choose a configuration on a frontier, not a model.

The cheapest configuration that clears the quality gate is rarely the smallest model or the largest one

Predict first

A cascade's large model costs about ten times the small one per request. Raising escalation from 30% to 50% changes cost per 1,000 requests how?

Text description of the figure

Editable hypothetical prices, routing share, cache hit rate and context length feed a table and a plot of cost per 1,000 requests, latency and simulated quality, with the non-dominated configurations highlighted.

Three axes, three different drivers Overview

Cost
Input x input price + output x output price, every call, every retry.
Latency
First token: queue + reading the input. Last token: + output x time per token.
Quality
Only a golden set with critical slices shows it. Never on an invoice.
Choice
Constraints first, then the cheapest survivor on the frontier.

Which lever moves which meter Table

Lever	Cost	Time to first token	Total latency	Quality risk
Cache a stable prefix	Lower on the cached part	Lower	Slightly lower	None: output unchanged
Batch interface	Lower (documented)	Not interactive	Asynchronous	None; only for work nobody waits for
Fewer retrieved passages	Lower	Lower	Slightly lower	May drop evidence: re-run golden set
Cap output length	Lower	No change	Lower	May cut needed content
Cascade or router	Lower if escalation is low	Fast path unchanged	Tail = escalated path	Only as good as the acceptance check
Streaming	No change	First word sooner	No change	None
Larger model	Higher	Usually higher	Usually higher	Possibly better: must be shown

Cost per request, then per good answer Formula

\begin{gathered} c = \sum_{\text{calls}} \left(n_{in} p_{in} + n_{cache} p_{cache} + n_{out} p_{out}\right) \\[4pt] c_{good} = c / q \end{gathered}

$n_{in}, n_{out}$: uncached input and output tokens of one call
$n_{cache} p_{cache}$: cached-prefix tokens at the cheaper cached-read price
$\sum_{\text{calls}}$: every call in the request: retries, escalations, judges, tool loops
$q$: pass rate on the golden set; c_good is the cost of one acceptable answer

Multiply by 1,000 for the unit managers use. Every price must be labelled an assumption until re-read from the provider.

Cost per 1,000 requests under hypothetical prices (cap: USD 5.00) Chart

Large modelquality 0.94, over the cap
Large + cached prefixcaching saves 1.46: only 600 of 2,400 tokens cacheable
Small modelquality 0.86, fails the gate
Cascade, 30% escalatedquality 0.93: the only pass

Illustrative values: they show the pattern, not measured results.

Computed by the application code from hypothetical prices and speeds; quality is simulated.

Constraints first: USD 5 per 1k, p95 7 s, pass rate 0.92 Comparison

Criterion	Large	Large + cache	Small	Cascade
Cost per 1k requests	10.20	8.74	0.85	3.47
p95 total latency	4.22 s	4.08 s	2.16 s	6.24 s (escalated path)
Pass rate	0.94	0.94	0.86	0.93
Cost per 1k good answers	10.85	9.30	0.99	3.73
Verdict	Fails cost	Fails cost	Fails quality	Passes; monitor escalation

Hypothetical prices and speeds, simulated quality. A configuration that is worse on every axis than another is dominated and drops out first.

One knob, two shapes: cost is a line, the p95 is a step Illustration

Every escalated request pays for both calls: 0.85 + share x 8.74 per 1,000. The cap is crossed at 47%; the cascade only costs more than the cached large model alone above 90% (off this chart).

Mean latency drifts up in a straight line. The p95 is 2.5 s at 2% escalated and 6.3 s at 10%: once more than 5% take the slow path, it sets the p95. At 30% the mean is still only 3.4 s.

Hypothetical prices and speeds from the application block (fast path 2.16 s, escalated path 6.24 s). Latency spread is simulated: 2,000 seeded requests per share with log-normal noise (sigma 0.08), which rounds the corner of the p95 step.

Cost per 1,000 requests and the latency distribution of the application's cascade as the escalation share grows. Averages fall, the tail rises: once more than 5% of requests take the slow path, that path sets the p95, however fast the rest is.

Key terms: time to first token, prompt caching, cascade, quality-cost frontier, Cost per supported answer.
Cost = input x input price + output x output price + cached reads at the cached price, summed over every call; divide by the pass rate for cost per good answer.
Latency is about TTFT + (output tokens - 1) x time per output token. Streaming moves the first word, not the last.
Levers act on different meters: price (caching, batch), volume (context, history, output), routing (cascade), capacity (limits, self-hosting).
Every price in this lesson is hypothetical.

Assumptions

Prices, speeds and cache rules are inputs you replace with your provider's current values and your own measurements; the numbers here are hypothetical.
Quality comes from a golden set that matches live traffic, including the slices where cheaper configurations fail first.
A cascade's acceptance check is reliable and its own cost is small relative to the large model.
Load is steady enough that the cache hit rate and escalation share measured in a test persist in production.

Application and workplace decision · 10 min

Four configurations of a telecom operator's plan-and-billing assistant, used by call-centre agents, under hypothetical prices and speeds: large, large + cached 600-token prefix, small, and a cascade (small first, 30% escalated to the cached large model).
Quality is a simulated golden-set pass rate.
Constraints: at most $5.00 per 1,000 requests, p95 total latency at most 7 s, pass rate at least 0.92.
Before running, guess which configuration passes all three and which constraint rules out each of the others.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

# HYPOTHETICAL inputs: prices in USD per million tokens, speeds in tokens per second. Quality is SIMULATED.
PRICE = {"large": (3.00, 15.00, 0.30), "small": (0.25, 1.25, 0.025)}   # input, output, cached-input read
SPEED = {"large": (4000, 60), "small": (12000, 120)}                  # prefill, decode
IN_TOK, STATIC, OUT_TOK, HIT, ESC = 2400, 600, 200, 0.9, 0.30         # per request; escalation share
CAP_COST, CAP_P95, MIN_Q = 5.0, 7.0, 0.92                              # constraints: USD/1k, seconds, pass rate

def cost(m, cache):
    p_in, p_out, p_hit = PRICE[m]
    static = STATIC * (HIT * p_hit + (1 - HIT) * p_in) if cache else STATIC * p_in
    return (static + (IN_TOK - STATIC) * p_in + OUT_TOK * p_out) / 1e6

def timing(m, cache):                                   # (time to first token, total) in seconds
    prefill, decode = SPEED[m]
    ttft = 0.3 + (IN_TOK - (STATIC * HIT if cache else 0)) / prefill
    return ttft, ttft + (OUT_TOK - 1) / decode

(l_t, l_T), (s_t, s_T), (lc_t, lc_T) = timing("large", False), timing("small", False), timing("large", True)
rows = {  # name: ($ per request, ttft, p95 total, simulated pass rate)
    "large":       (cost("large", False), l_t, l_T, 0.94),
    "large+cache": (cost("large", True), lc_t, lc_T, 0.94),
    "small":       (cost("small", False), s_t, s_T, 0.86),
    "cascade":     (cost("small", False) + ESC * cost("large", True), s_t, s_T + lc_T, 0.93),  # p95 = escalated path
}
for name, (c, t, p95, q) in rows.items():
    ok = c * 1000 <= CAP_COST and p95 <= CAP_P95 and q >= MIN_Q
    print(f"{name:12s} ${c*1000:5.2f}/1k  ttft {t:.2f}s  p95 {p95:.2f}s  quality {q:.2f}  "
          f"${c*1000/q:5.2f} per 1k good  {'PASS' if ok else 'fail'}")
room = (CAP_COST / 1000 - cost("small", False)) / cost("large", True)
print(f"cascade stays under the cost cap up to {room:.0%} escalation (assumed {ESC:.0%})")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 30 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Cost per request, decomposed term by term6 minThe cost of one task as a sum over calls, attempts and failure handling, worked by hand on a claims-desk assistant and recomputed in Python.

From one prompt to a resolved request: the token bill is 3% of it Bridge

Static prefix, 80% hits0.28 USD/1k
Claim file + question3,000 uncached input tokens+3 USD/1k
Answer400 output tokens+2 USD/1k
Retries, E[A] = 1.0525f = 0.05, m = 3+0.28 USD/1k
Verifier calloften forgotten+0.7 USD/1k
Failed answers to a handler(1 − q) × h = 0.10 × USD 2.00+200 USD/1k
Cost per resolved request206.26 USD/1k

Claims-desk example per 1,000 requests, hypothetical prices (the worked table below). Each step is a measured count times a price re-read from the provider; retries enter through E[A] = (1 − f^m)/(1 − f) from Day 52. A forgotten step (verifier, retries, escalation) is the usual reason a forecast misses the invoice.

Cost of one task, every term written out Formula

\begin{gathered} c_{task} = \sum_{j} \pi_j\, \mathbb{E}[A_j]\, c_j + (1 - q)\, h \\[4pt] c_j = n^{in}_j p_{in} + n^{cache}_j p_{cache} + n^{out}_j p_{out} \end{gathered}

$j$: each kind of call a request can make: generator, verifier, escalation to a larger model, embedding
$\pi_j$: share of requests that make call j: 1 for the generator, the escalation share s for the large model
$\mathbb{E}[A_j]$: expected attempts of call j, (1 - f^m)/(1 - f) from Day 52; an upper bound on billing, since a request refused before work is not billed
$c_j$: the token bill of one attempt of call j: uncached input, cached reads and output, each at its own price
$q$: share of requests answered acceptably, measured on the golden set; the token sum divided by q is the token cost per good answer
$h$: cost of handling one failed answer outside the invoice: a person's time, a repeat contact, a correction

Cost is linear in every count and share, so each coefficient is a sensitivity: one more point of escalation costs c_large, one more point of pass rate saves h. Units shortcut: tokens per request x USD per million tokens / 1,000 = USD per 1,000 requests.

Worked by hand: an insurance claims-desk assistant, per 1,000 requests (hypothetical) Table

Term	Computation	USD per 1,000 requests
Prices (assumed)	input 1.00, cached read 0.10, output 5.00 USD per million tokens	-
Static prefix	1,000 tokens x (0.8 x 0.10 + 0.2 x 1.00) / 1,000, at 80% cache hits	0.28
Claim file + question	3,000 uncached tokens x 1.00 / 1,000	3.00
Answer	400 output tokens x 5.00 / 1,000	2.00
One generator attempt	0.28 + 3.00 + 2.00	5.28
Retries, f = 0.05, m = 3	E[A] = (1 - 0.05^3) / 0.95 = 1.0525; 5.28 x 1.0525	5.56
Verifier call	(600 input x 1.00 + 20 output x 5.00) / 1,000	0.70
Token bill	5.56 + 0.70	6.26
Per good answer, q = 0.90	6.26 / 0.90	6.95
Failed answers to a claims handler	0.10 x USD 2.00 x 1,000 (assumed handling cost)	200.00
Cost per resolved request	6.26 + 200.00	206.26

Recomputed in Python (simulated E[A] = 1.052 over 200,000 requests, numpy default_rng(0)). The token bill is 3% of the total: one point of pass rate is worth USD 20 per 1,000 requests here, more than three times the whole token bill. Assumes every failed answer is caught and handed to a person.

Rank levers by coefficient × movable amount: here quality dwarfs caching Chart

Pass rate +1 point0.01 × 2.00 handling × 1,00020 USD/1k
No retries (f 0.05 → 0)5.28 × 0.05250.28 USD/1k
Cache hits 80% → 100%prefix term 0.28 → 0.100.18 USD/1k

Savings per 1,000 requests on the claims-desk example (hypothetical). The linear form breaks in an incident: failure rate, retries and escalation share rise together (Day 52), so the counts are not independent; price the incident week as its own scenario.

The p95 of a mixture, and why it stepsadvanced6 minWhy a percentile of two latency paths jumps when the slow share passes 1 - p, derived from the mixture CDF and worked by hand on an e-commerce cascade.

The p95 of a two-path mixture Formula

\begin{gathered} F(t) = (1-s)\,F_{f}(t) + s\,F_{e}(t) \\[8pt] t_{95} =\begin{cases} F_{f}^{-1}\!\left(\dfrac{0.95}{1-s}\right) & s < 0.05 \\[6pt] F_{e}^{-1}\!\left(1 - \dfrac{0.05}{s}\right) & s > 0.05 \end{cases} \end{gathered}

$F_f,\ F_e$: latency distribution (CDF) of the fast path and of the escalated path, each measured separately
$s$: share of requests escalated: the mixture weight
$F(t)$: share of ALL requests finished by time t
$t_{95}$: the 95th percentile: the smallest t with F(t) >= 0.95
$F^{-1}(u)$: the u-quantile of one path
$\text{validity}$: exact when the paths do not overlap (every fast request ends before any escalated one), as when escalation is a fast attempt plus a large call

The same algebra for any percentile p puts the switch at s = 1 - p, so a p99 objective breaks at 1% escalation. Report p50 and p95 for first token and for total time, per path.

Deriving the step, line by line Sequence

Write the mixture
Each request escalates with probability s, so the share finished by time t is the weighted sum of the two path CDFs.
Find the gap
If every fast request ends before any escalated one, F(t) is flat at 1 - s between the two paths.
s below 5%
1 - s > 0.95, so F crosses 0.95 inside the fast path: solve (1 - s) F_f(t) = 0.95.
s above 5%
F is still 1 - s < 0.95 across the gap, so the crossing is inside the escalated path: solve (1 - s) + s F_e(t) = 0.95.
The step
At s = 0.05 the solution jumps across the whole gap. The mean, (1 - s) mean_f + s mean_e, moves smoothly, which is why it hides the jump.
Read the formula
At s = 0.10 the p95 is the median of the escalated path (1 - 0.05/0.10 = 0.5); at s = 0.25 it is that path's 80th percentile.

Worked by hand: an e-commerce support cascade (illustrative uniform paths) Table

Escalated share s	Path holding the p95	Quantile within that path	p95 by hand	Mean
0.02	fast	0.95 / 0.98 = 0.969	0.6 + 0.6 x 0.969 = 1.18 s	0.95 s
0.04	fast	0.95 / 0.96 = 0.990	0.6 + 0.6 x 0.990 = 1.19 s	1.00 s
0.06	escalated	1 - 0.05 / 0.06 = 0.167	2.8 + 1.2 x 0.167 = 3.00 s	1.05 s
0.10	escalated	1 - 0.05 / 0.10 = 0.500	2.8 + 1.2 x 0.500 = 3.40 s	1.15 s
0.30	escalated	1 - 0.05 / 0.30 = 0.833	2.8 + 1.2 x 0.833 = 3.80 s	1.65 s

Fast path uniform on 0.6-1.2 s, escalated path uniform on 2.8-4.0 s (hypothetical shapes, chosen so quantiles are linear); mean = 0.9 + 2.5 s. Simulation (numpy default_rng(1), 400,000 requests per row) gives p95 1.182, 1.194, 3.003, 3.396 and 3.800 s. From 4% to 6% escalated the mean moves 0.05 s and the p95 1.8 s.

The p95 step belongs to any two-path latency, not to cascades Overview

Cascade
Fast answer, or small attempt plus large call.
Cache hit or miss
A hit skips the prefix read; a miss pays it in full.
Retried or not
A retried request adds a backoff wait and a second call.
Cold or warm start
A cold replica or a tool call adds a fixed chunk of seconds.
Diagnostic
Report p95 and share per path; alert on the share, which moves before the objective breaks.

With per-path p95 and share, the overall p95 is predictable from the mixture formula. Overlapping, noisy paths round the corner off, but the jump survives (the cascade demo uses log-normal noise).

Latency, decomposed3 minThe formula for first token and total time, and which part each lever shortens.

Where the seconds go Formula

\begin{gathered} T_{total} \approx \mathrm{TTFT} + (n_{out} - 1)\,\mathrm{TPOT} \\[4pt] \mathrm{TTFT} \approx t_{queue} + n_{in} / r_{prefill} \end{gathered}

$\mathrm{TTFT}$: time to first token: queueing plus reading the whole input
$\mathrm{TPOT}$: time per output token (1 / decode speed)
$r_{prefill}$: input tokens processed per second; cached tokens are not re-read

The application code uses exactly this split with a 0.3 s queue term. Both metrics are defined in the OpenTelemetry GenAI conventions (status Development, as read 2026-10-11).

Latency split under the hypothetical speeds (2,400 in, 200 out) Chart

Large, first token
Large, generation
Large + cache, first tokencache trims the input read only
Large + cache, generation
Small, first token
Small, generation

Illustrative values: they show the pattern, not measured results.

Generation is 199 tokens at 60 or 120 tokens per second. Caching shortened the first token by 0.135 s and left generation untouched.

Caching, on the part that repeats3 minWhere the money goes inside one request and how to make a prefix cache actually hit.

Large model, cost per 1,000 requests by part (hypothetical prices) Chart

Static prefix, uncached
Static prefix, 90% cache hits
Retrieved context and question1,800 tokens: the biggest part, not cacheable
Output (200 tokens)

Illustrative values: they show the pattern, not measured results.

Caching works on the part that repeats. Here the per-request context dominates, so trimming passages moves cost more than caching.

What a prefix cache can save, and why the hit rate is not yours to choose Illustration

600 of 2,400 repeat (the application)
1,200 repeat
1,800 repeat

At a perfect hit rate the cache saves 1.62 per 1,000 when 600 tokens repeat and 4.86 when 1,800 do. The slope is the cacheable share; the rest of the bill does not move.

1 shared prefix
10 variants (per tenant, per language)
100 variants

The hit rate is not a setting: with random arrivals it is 1 - exp(-requests per lifetime / variants). Splitting one prefix into many variants divides the traffic each one sees.

Hypothetical prices from the application block (input 3.00, cached read 0.30, output 15.00 per million tokens); cache-write premiums ignored. Hit rates assume random (Poisson) arrivals and an entry that lives one lifetime after its last use; circles are 3,000 seeded simulated arrivals each. No provider's cache lifetime or minimum size is assumed: read yours and measure hits from response token counts.

The saving is the repeated share of the input times the hit rate; the hit rate is set by traffic per cache lifetime and by how many prefix variants split it.

Making a prompt cache hit (one provider's documented mechanism, read 2026-10-11) Sequence

Stable content first
Tools, then system prompt, then messages; a change invalidates that level and all after it.
Identical prefix
Any edit to an early tool definition or system line turns hits into misses.
Long enough
Prompts below a model-specific minimum are not cached at all.
Short lifetime
Entries expire after a short idle time, so low traffic gets few hits.
Measure it
Read cache-read and cache-write token counts in responses; never assume a hit rate.

Cascades and routers4 minThe decision path; the linear cost line and the p95 step are drawn in the concept block.

How a cascade decides Diagram

Request
every request
Small modelcheap, fast
draft answer
Passes the check?schema, confidence
- yes: to Answer
no: escalate
Large modelpays for both calls
- slow path: to Answer
Answerfast path
Answerescalated path

A router picks a model before calling any. Both need the threshold tuned on one sample and confirmed on another. FrugalGPT (2023 preprint) reports large savings on its authors' tasks; treat headline savings as task-specific.

Managed versus self-hostedadvanced3 minWhy self-hosting has a fixed floor, where break-even sits under the lab's hypothetical numbers, and what to measure first.

Managed API versus two self-hosted nodes (hands-on lab, hypothetical) Chart

Managed, USD 0.0102 per request
Self-hosted, nodes + people

Illustrative curves: they show the pattern, not measured results.

Self-hosting starts at a fixed floor (two nodes for failover plus people) and wins only above the break-even volume. Measure throughput before trusting it.

Self-hosted cost per token is set by utilisation Formula

c_{token} = \frac{P_{node}}{3600 \cdot r \cdot u}

$P_{node}$: node price per hour (hypothetical), before people and the second node for failover
$r$: tokens per second at full load, measured; server-side batching raises it at the price of waiting for the batch
$u$: utilisation: share of the hour the node is busy, set by traffic shape and batching

A node at 25% load costs four times as much per token as at 100%. Server-side batching is not the provider's batch interface: it is your server trading per-request wait for throughput.

Reliability and capacity2 minDependencies in series, fallbacks that fail together, and why capacity runs out faster than traffic grows.

Reliability is a cost line too Overview

Dependencies in series
Availability is bounded by every service a request needs.
Independent fallback
A second model at the same provider and region may fail together.
Degraded modes
Cached answer, retrieval-only result or human handoff: usually cheaper than a second path.
Limits are capacity
Tokens per minute cap throughput; one provider excludes cache reads from input limits for most models (as read).

Capacity near saturation, and why big pools run hotter Illustration

Eight concurrent slots: at 60% load the p95 wait is 1.4 s, at x1.3 traffic 5.1 s, at x1.5 13.9 s. The last 30% of traffic costs more waiting than the first 60%; a retry storm is exactly such a multiplier.

Slots needed for a p95 wait of at most 2 s. Sixty-four times the traffic needs 24 times the slots: large pools run at 92% load where a small one manages 35%.

Queueing model, not a measurement: random (Poisson) arrivals, exponential service with a 4.22 s mean (the application block's hypothetical large-model latency), c identical slots. Real LLM servers batch requests and their service time depends on tokens, so measure your own throughput curve; the shape near saturation is what carries over.

A queueing model of concurrent slots. Waiting grows steeply as load nears capacity; the slots a fixed wait needs grow slower than traffic. Plan headroom against incident traffic (retries, re-sends), not the daily average.

Conversations, assumptions and what to monitoradvanced3 minWhy chat cost grows faster than the number of turns, and the assumption behind each saving that must stay on a dashboard.

Chat history: cumulative input tokens over a conversation Chart

Full history resent every turn
Last 4 turns kept

Assumed: 1,000-token system prompt, 60-token user turns, 200-token answers. Full history grows quadratically in the number of turns; a window grows linearly.

Every saving rests on an assumption; monitor the assumption Table

Saving	Rests on	Signal on the dashboard	Re-evaluate when
Prefix cache	Hit rate on a stable prefix	Cache-read tokens / cacheable tokens	A tool or system line changes
Cascade or router	Escalation share and check accuracy	Escalation share per slice; p95 per path	Traffic mix shifts or threshold moves
Shorter context	Evidence still in the top k	Recall@k on a labelled sample	Corpus or chunking changes
Output cap	Answers still complete	Finish reason = length rate	Prompt or task changes
Self-hosting	Utilisation at the latency target	Load and p95 per node	Traffic shape changes

Practitioner checklist. Each row names what to recompute before claiming the saving again.

Memory anchor

Price the good answer, split the wait, and monitor the assumption each saving rests on.

Useful for

Turning "the assistant is too expensive" into a cost per 1,000 requests and cost per good answer with named assumptions, for a SaaS support bot or a bank's document-extraction pipeline alike.
Choosing between caching, routing, shorter context and a different model on evidence instead of habit.
Setting latency targets that separate time to first token from total time.

Not suitable for

Quoting list prices or provider limits as facts; those change, and every number here is hypothetical.
Tasks with a hard quality floor where only the strongest model qualifies and cost is secondary, such as a draft a clinician must sign.

Traps

Price per token as the cost of a feature

The estimate ignores retries, escalations, long chat history and human handling of failed answers, so the real bill is higher and unexplained. A marketing team pricing a campaign-copy generator per token finds regenerations and human edits are most of the cost.

How to avoid it

Compute cost per successful task from measured tokens per attempt and attempts per task, with every price labelled an assumption.

Averages that hide the tail

A cascade or retry policy lowers mean latency while the p95 path gets longer, and users on that path see the slowdown. In a logistics ETA chatbot, the few queries that call the routing tool set the p95.

How to avoid it

Report p50 and p95 for time to first token and for total time, per path.

Senior reasoning

A senior asks what the unit of value is and prices that: a good answer, not a token. They split latency into the parts the user feels, evaluate configurations on one golden set, name the assumption each saving rests on (cache hit rate, escalation share, volume) and put that assumption on a dashboard.

Interview question

A media company's article-summarisation feature tripled its monthly bill after launch and its p95 latency doubled. How do you investigate and what would you change first?

Outline your answer first.

Recall cards scheduled after completion

Write the cost-per-1,000-requests formula and name the three assumptions it rests on.
Which part of latency do input length, output length and streaming each change?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance — Lingjiao Chen, Matei Zaharia, James Zou, 2023core
Preprint on prompt adaptation, model approximation and LLM cascades; reports that API prices differed widely and that a learned cascade matched a strong model at much lower cost on the authors' tasks.
Prompt caching (Claude Platform Docs) — Anthropiccore
One documented caching design - identical prefix, tools then system then messages, cache reads billed at a fraction of base input, lower time to first token.
OpenTelemetry semantic conventions for generative AI (spans and metrics) — OpenTelemetryoptional
The registered spans page is status Development; the metrics page of the same convention set defines time to first token and time per output token as server metrics, also Development, so names may change.
Batch processing (Claude Platform Docs) — Anthropicoptional
Asynchronous batch interface documented as lower cost and higher throughput for work that does not need an immediate answer.
Rate limits (Claude Platform Docs) — Anthropicoptional
Limits counted in requests, input tokens and output tokens per minute, with most cache reads excluded from input-token limits.
Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023optional
Key-value cache memory limits batch size and throughput in self-hosted serving, which is why utilisation drives per-token cost.
LLM10:2025 Unbounded Consumption (OWASP Top 10 for LLM Applications) — OWASP Gen AI Security Project, 2025optional
Denial of wallet and the quotas, input limits and timeouts that bound consumption.

Optional sections

Deep dive · 10 min

Cost structure. Input tokens usually dominate in RAG (long retrieved context) and output tokens dominate in generation-heavy tasks; price tables differ, so compute both. Multi-turn chat resends history, so input cost per turn grows with conversation length and total cost grows faster than the number of turns. Cap history, summarise older turns or keep only task state, and cache the stable prefix.

Caching. A provider cache needs an identical prefix; changing a tool definition or reordering the system prompt invalidates it. Check cache read and write token counts in responses rather than assuming a hit rate. Below some minimum prompt size a prefix may not be cached at all, and cache lifetimes are short, so low traffic gets few hits.

Batching, two meanings. A batch API runs non-urgent work asynchronously at a lower price. Server-side batching groups concurrent requests on a GPU to raise throughput, at the price of waiting for the batch. Both raise throughput; only the first is a pricing lever you control.

Cascades and routers. A cascade needs an acceptance signal (schema validity, retrieval confidence, a verifier). A router chooses a model before calling any. Both need a golden set that includes the slices where the small model fails; ensure the threshold is tuned on one sample and confirmed on another. FrugalGPT and RouteLLM are preprints: treat the headline savings as task-specific.

Managed versus self-hosted. Per-token cost of a GPU you run is the node price divided by the tokens it serves, so utilisation (traffic shape and batching) dominates. Add engineers, on-call, evaluation and upgrade work. Managed pricing includes idle capacity, which is why it wins at low or spiky volume. Small embedding and reranking models may run acceptably on CPU; measure p95 on your hardware before deciding.

Reliability. Feature availability is bounded by its dependencies in series. A fallback model helps only if its failures are independent of the primary's; a second model at the same provider and region may fail together. Degraded modes (cached answer, retrieval-only result, human handoff) are usually cheaper than a second full path.

Hands-on lab: Break-even between a managed API and a self-hosted node (all numbers hypothetical) · 15 min

The code compares a managed API with a self-hosted GPU deployment (at least two nodes, one as failover) for the same workload under hypothetical numbers, and prints the break-even volume. Run it, then (1) halve the node throughput and find the new break-even, (2) set the people cost to zero and say how much of the decision was people, and (3) say which assumption you would measure first before trusting the result.

import math

# HYPOTHETICAL numbers throughout; replace them with measured throughput and your own prices.
managed_per_req = 0.0102             # USD per request on a managed API (large model, no cache)
node_per_hour, hours = 2.50, 730     # USD per GPU node-hour, hours per month
people = 4000.0                      # USD per month: share of on-call, upgrades, evaluation
node_rps, target_util = 2.0, 0.7     # sustained requests/s per node within the latency target; headroom
node_cap = node_rps * 3600 * hours * target_util      # requests per month one node serves

def self_hosted(volume):
    nodes = max(2, math.ceil(volume / node_cap))      # at least two nodes, so one can fail
    return nodes, nodes * node_per_hour * hours + people

print(f"one node serves {node_cap:,.0f} requests per month at {target_util:.0%} utilisation")
for volume in (200_000, 500_000, 1_000_000, 4_000_000, 10_000_000):
    nodes, hosted = self_hosted(volume)
    managed = volume * managed_per_req
    print(f"{volume:>10,}/month  managed ${managed:>9,.0f}  self-hosted ${hosted:>8,.0f} ({nodes} nodes)  "
          f"{'self-hosted cheaper' if hosted < managed else 'managed cheaper'}")
be = next(v for v in range(10_000, 20_000_000, 10_000) if self_hosted(v)[1] <= v * managed_per_req)
print(f"break-even volume (first where self-hosted costs no more): {be:,} requests per month")

Interview practice · 10 min

A retail bank's CFO asks you to halve the cost of its LLM document-extraction feature without hurting quality. What is your plan?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 30 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min