Core day 55 of 60 · Week 11 · Part II · evaluation depth

Production GenAI Case

You must defend a production design for a telecom operator's plan-and-billing assistant in front of engineering, security and finance. Three candidates exist, none is perfect, and the launch date is fixed. How do you decide, and what do you write down so the decision can be revisited?

By the end: Given gates, three candidate designs with labelled simulated evidence and two shocks, you decide what to pilot, say which constraint you would negotiate and which you would not, and write a one-page decision record with numeric revisit triggers.

At a glance

Fix the gates with their owners first, test every candidate against them with labelled evidence, and end with what you give up and your revisit triggers.

Gates before results
Quality, abstention, latency, cost, security, capacity and a degraded mode, agreed by owners.
Real alternatives
Include the prototype and a non-generative option such as retrieval-only passages.
Label every number
Measured, simulated or hypothetical, in the table and in the record.
Budgets negotiate
Cost and latency can move with a business case; correctness floors and data rules do not.
Triggers as numbers
Escalation share, sample size, traffic, retirement date: each with an owner.

Recall · 5 min

Without notes: write the cost-per-1,000-requests formula (Day 54), one rule for retries on a write (Day 52), and one control that holds even if the model obeys injected text (Day 53).

Open due reviews

Concept · 10 min

A design review is a negotiation between constraints that each have an owner: quality, security, cost and reliability. A design that wins on one and ignores the others is a prototype. Your job is to make the trade-offs explicit enough to defend and honest enough to revise.

A design is a point on the quality, cost and latency surface, and the gates decide which points are allowed

Predict first

Traffic rises 2.5 times. Design B sends every request to the large model with a cached prefix. Which gate does it fail first?

Text description of the figure

Editable hypothetical prices, routing share, cache hit rate and context length produce cost per 1,000 requests, latency and simulated quality for each configuration, so you can see which configurations a quality floor allows and how price, routing or traffic moves their cost. The lab has no tokens-per-minute or security gate; those come from the case code.

Four constraints, four owners Overview

Quality
Product owner and the people who receive wrong answers: correct, supported, abstains when it should.
Security
Whoever signs off data access and tools: attack suite, enforceable controls.
Cost
Finance: cost per 1,000 requests and per month at expected traffic.
Reliability
On call: p95, capacity at peak, a degraded mode that was drilled.

The gate-first defence Sequence

Frame the decision
Who acts on the answer, what a wrong answer costs, what cannot answer means.
Fix the gates
Thresholds and evidence type per gate, signed before any candidate result is shown.
List real alternatives
The prototype, a non-generative option, and one cost lever such as caching or a cascade.
Attach labelled evidence
Each number marked measured, simulated or hypothetical.
Decide and give up
State what the decision sacrifices and who accepted it.
Write triggers
Numbers with owners: escalation share, reviewed sample, traffic, retirement date.

Gates set after the results are not gates: a threshold chosen after seeing the candidates lets the favourite design define success. Owners disagree less about numbers they fixed before anyone had a favourite.

Three candidates against the signed gates (simulated, hypothetical prices) Comparison

Criterion	A prototype	B rerank + cache	C cascade 30%
Faithful answers (gate 0.92 of 60)	50 of 60: fail	56 of 60: pass	55 of 60 (0.917): fail by one answer
Correct abstentions (gate 15 of 20)	9: fail	15: pass	15: pass
p95 (gate 5 s)	5.6 s: fail	3.6 s: pass	4.8 s: pass
Cost per 1k (gate USD 5)	13.80: fail	6.94: fail	2.78: pass
Attacks blocked (gate 11 of 12)	5: fail	11: pass	11: pass
Peak tokens/min (limit 400k)	601k: fail	200k: pass	60k: pass
Degraded mode	None: fail	Retrieval-only: pass	Retrieval-only: pass

Output of the case code at today's traffic. Each failed cell is a candidate for a specific experiment, not for a vote.

Which failed gate is a negotiation? Diagram

A gate fails
classify it
Budget gate? (cost, p95)
- yes: to Negotiate
no
Do not move itcorrectness floor, data rule, law
- redesign: to Narrow or redesign
Negotiatebusiness case to the budget owner
- if accepted: to Record who accepted it
Narrow or redesignsmaller launch
Record who accepted it

In the case, B's cost gap goes to the budget owner; C's quality gap goes to a larger sample, never to a lower gate.

Six steps: frame the decision; fix the gates before results; list real alternatives (including a non-generative one); attach labelled evidence; decide and say what you give up; write revisit triggers as numbers.
Every number is labelled measured, simulated or hypothetical.
Reliability needs a tested Degraded mode; security needs an attack suite with enforceable controls (Day 53).
Two challenges recur: "use the largest model" (it must meet the gates at the cost cap) and "your set is too small" (enlarge the sample on the gate that decides).
The output is a one-page decision record. Scenarios S07, S05 and S23 practise the same moves.

Assumptions

Owners agree the gates and thresholds before candidate results are shown, and they are not renegotiated to fit a favourite design.
Every number keeps its label, and prices and limits are re-read from provider documentation before sign-off.
The golden set and attack suite represent live traffic and attackers well enough to decide on, and both grow after launch.
The degraded mode exists and was tested, because otherwise availability depends entirely on the provider.

Application and workplace decision · 10 min

The assistant serves 120 call-centre agents who quote tariffs and fees to customers. Hypothetical workload: 40,000 requests a day, peak 167 a minute.
1. Frame. Write your gates before running anything; the GATE line is the version the owners signed.
2. Options. A, the demo prototype; B, hybrid retrieval, rerank to three passages, cached prefix; C, B behind a cascade sending 30% to the large model.
3. Shock one. The code re-runs at 2.5 times traffic for a planned self-service channel.
4. Shock two. The provider announces retirement of the large model in 90 days.
Then write the decision, what you give up and your revisit triggers. Measurements are simulated; prices and limits are hypothetical.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

# SIMULATED measurements, HYPOTHETICAL prices (USD per million tokens) and limits. Replace with your own.
PL, PS = (3.00, 15.00, 0.30), (0.25, 1.25, 0.025)                    # large / small: input, output, cached read
GATE = dict(quality=0.92, abstain=15, p95=5.0, cost=5.0, attacks=11, itpm=400_000)   # cost: USD per 1k requests
PEAK_RPM, OUT, DAILY = 167, 200, 40_000                               # peak requests/min, output tokens, requests/day
# name: (input tokens, cached prefix, share sent to large, faithful of 60, correct abstentions of 20, p95 s,
#        injection cases blocked of 12, degraded mode exists)
D = {
    "A prototype":    (3600, 0,   1.0, 50,  9, 5.6,  5, False),
    "B rerank+cache": (1800, 600, 1.0, 56, 15, 3.6, 11, True),
    "C cascade 30%":  (1800, 600, 0.3, 55, 15, 4.8, 11, True),
}

def unit(tok, cached):                                                # USD per 1,000 requests: (large, small)
    large = cached * (0.9 * PL[2] + 0.1 * PL[0]) + (tok - cached) * PL[0] + OUT * PL[1]
    return large / 1e3, (tok * PS[0] + OUT * PS[1]) / 1e3

for traffic in (1.0, 2.5):
    for name, (tok, cached, share, ok, ab, p95, atk, fb) in D.items():
        large, small = unit(tok, cached)
        cost = share * large + (small if share < 1 else 0.0)
        itpm = PEAK_RPM * traffic * share * (tok - cached)            # cached reads excluded from the limit (assumed)
        bad = [k for k, v in {"quality": ok / 60 < GATE["quality"], "abstain": ab < GATE["abstain"],
               "p95": p95 > GATE["p95"], "cost": cost > GATE["cost"], "attacks": atk < GATE["attacks"],
               "capacity": itpm > GATE["itpm"], "no fallback": not fb}.items() if v]
        print(f"traffic x{traffic}: {name:15s} ${cost:5.2f}/1k  peak {itpm/1000:4.0f}k tok/min  "
              f"quality {ok}/60  {'PASS' if not bad else 'fails ' + ', '.join(bad)}")

large, small = unit(1800, 600)
print(f"B monthly: ${large*DAILY*30/1000:,.0f} vs cap ${GATE['cost']*DAILY*30/1000:,.0f}; "
      f"C can escalate up to {(GATE['cost'] - small) / large:.0%} within the cost cap")

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 23 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

Capacity budgets and break-even algebra, as a method7 minTokens-per-minute budgets and break-even volumes as one linear method, worked by hand in logistics and energy, plus the sensitivity that says what to measure.

Size against the peak minute: six times the average Chart

Average minute (40,000 / 1,440)what a daily total suggests27.8 requests/min
Measured peak minutewhat the rate limit sees167 requests/min

The case's hypothetical workload. Every capacity and cost question here is the same move (the method steps below): write the quantity as a line in one driver, set it equal to a limit, solve. Use the peak minute as the rate, never the daily total divided by 1,440.

Tokens-per-minute budget and the traffic multiple that breaks it Formula

\begin{gathered} \mathrm{TPM}_{in}(g) = \lambda_{peak}\, g\, s\, (n_{in} - n_{cache}) \\[4pt] g^{*} = \min_{k} \frac{L_k}{\mathrm{TPM}_k(1)} \end{gathered}

$\lambda_{peak}$: requests per minute in today's peak minute, measured; not the daily total divided by 1,440
$g$: the traffic multiple being tested: 1 today, 2.5 for a new channel, the seasonal peak
$s$: share of requests that reach the rate-limited model: 1 for a single model, the escalation share for a cascade
$n_{in} - n_{cache}$: input tokens that count against the limit when cached reads are excluded (one provider's documented rule; confirm yours)
$L_k$: each limit on that model: input tokens, output tokens and requests per minute
$g^{*}$: the traffic multiple at which the FIRST limit binds; it becomes the capacity trigger in the record

The same template budgets output tokens (lambda g s n_out) and requests (lambda g s); the smallest ratio binds first. Caching relieves only the input ratio, so the binding limit can switch: recompute every ratio after each lever. Linearity is the assumption: a new channel can change s and n (longer conversations, a new traffic mix) as well as g.

Worked: a logistics delivery-exception assistant (hypothetical limits) Table

Quantity	Computation	Value
Today's peak	240 requests a minute; 40% escalate to the large model	lambda = 240, s = 0.4
Uncached input per large request	3,000 input tokens - 1,200 cached prefix	1,800 tokens
Input TPM at g = 1	240 x 0.4 x 1,800	172,800
Headroom on a 300k input limit	300,000 / 172,800	g* = 1.74
If cached reads counted	240 x 0.4 x 3,000 = 288,000; 300,000 / 288,000	g* = 1.04
Output TPM, 250 tokens each	240 x 0.4 x 250 = 24,000 against an 80k limit	g* = 3.33
Binding limit	min(1.74, 3.33)	input tokens, at 1.74x

Recomputed in Python. The cache rule alone moves the trigger from 4% growth to 74% growth, which is why it is re-read from the provider, never assumed. A seasonal peak (holiday parcels) is a traffic multiple g: check it before the season.

Break-even between two cost lines, and how sensitive it is Formula

\begin{gathered} C_i(x) = F_i + v_i\,x \;\Rightarrow\; x^{*} = \frac{F_2 - F_1}{v_1 - v_2} \\[4pt] \frac{\Delta x^{*}}{x^{*}} \approx -\frac{\Delta(v_1 - v_2)}{v_1 - v_2} \end{gathered}

$F_i$: fixed monthly cost of option i: reserved capacity, nodes, the people who run them
$v_i$: variable cost per request, from the Day 54 cost formula (tokens x prices, attempts, escalations)
$x$: the driver, here requests per month; the algebra is the same for any driver
$x^{*}$: the volume at which both options cost the same; above it the option with the lower v wins
$\Delta x^{*}/x^{*}$: to first order, the relative change of x* equals minus the relative change of the GAP v1 - v2: close variable costs make x* fragile

The case's escalation cap is the same algebra with the share as the driver: s_max = (cap - c_small) / c_large.

Worked: an energy retailer's billing assistant, pay-per-use vs reserved (hypothetical) Table

Step	Computation	Result
Option 1: pay per request	F1 = 0, v1 = USD 0.0060	-
Option 2: reserved capacity	F2 = USD 9,000 a month, v2 = USD 0.0015	-
Break-even volume	x* = (9,000 - 0) / (0.0060 - 0.0015) = 9,000 / 0.0045	2.00 million a month
Sensitivity: v1 falls 20%	9,000 / (0.0048 - 0.0015) = 9,000 / 0.0033	2.73 million (+36%)
Sensitivity: v2 rises to 0.0020	9,000 / (0.0060 - 0.0020) = 9,000 / 0.0040	2.25 million (+13%)
A cost cap as a break-even	cap 4.00, small 0.60, large 8.50 per 1,000: (4.00 - 0.60) / 8.50	s_max = 40% escalation

Recomputed in Python. A 20% price cut on the pay-per-use side moves the break-even by 36%, because the gap shrinks from 0.0045 to 0.0033 (the first-order formula gives 27%; the exact change is larger). State a break-even in the record as a trigger with the forecast beside it, never as a verdict.

Capacity budgets and break-evens are one method Sequence

Name the driver
Requests per minute, requests per month, escalation share or traffic multiple: the one quantity the decision turns on.
Write each side as a line
Load = rate x share x counted tokens; cost = fixed + variable x volume. Each coefficient is a measured count or a dated price.
Solve at the limit
Set the line equal to the limit or the other option; solve for the driver: g* (rate limit), x* (break-even), s_max (cost cap).
Put the forecast beside it
Write the forecast and its range next to the threshold. A forecast close to the threshold means the decision rests on the forecast's error.
Move each coefficient
Recompute with each coefficient at its plausible error; the one that moves the threshold most is the one to measure first.
Record it as a trigger
The threshold becomes a numeric revisit trigger in the decision record, with an owner and the signal that tracks it.

Defending under challenge3 minFour challenges you will hear, what each one tests, and an answer that concedes the valid part.

Answering the standard challenges Table

Challenge	What it really tests	Answer with
Use the largest model, to be safe	Whether quality is the only gate	Add it as a candidate; it must pass every gate at the cost cap
Your evaluation set is too small	Whether you know your uncertainty	Concede; enlarge the sample on the gate that decides, not on all
Put all documents in the prompt	Whether retrieval earns its complexity	Test it: cost growth, mid-context accuracy, access control, limits
Lower the gate, it was arbitrary	Whether the gate protects anyone	Keep it; record who would change it and why, in a new record

How precise is a reviewed pass rate near 0.92? (95% half-width) Chart

60 reviewed answersone answer = 1.7 points6.9 points
150 reviewed answers4.3 points
300 reviewed answers3.1 points
600 reviewed answers2.2 points

Normal approximation, 1.96 x sqrt(0.92 x 0.08 / n). C's 0.917 against 0.92 is far inside the 60-answer interval; compare B and C on paired questions.

How often a review of n answers passes a design, given its true quality Illustration

60 reviewed answers
150 reviewed answers
600 reviewed answers

Exact binomial probabilities, assuming reviewed answers are independent draws from live traffic. Left of 0.92 the percentage is the chance of wrongly passing; right of it, the chance of passing a design that deserves it.

An ideal gate would be a step at 0.92: always pass a design above it, never one below. A 60-answer review is far from that step; a few hundred reviewed answers start to separate 0.90 from 0.94.

The operating characteristic of the 0.92 gate. With 60 answers, designs well below the gate pass often and designs above it fail often.

What goes into the security gate2 minInputs, tools, identity, approvals, the attack suite and consumption limits.

What the security gate contains Sequence

Untrusted inputs
Every source of text the model reads, including tool results.
Tools and identity
Per-task allow-lists; the user-scoped identity each call runs under.
Approvals and output
Approval points for irreversible actions; link and content controls on output.
Attack suite
Pass threshold agreed in advance; residual cases listed with an owner.
Consumption
Quotas and spend limits: cost is also an attack surface.

What goes into the reliability gate3 minCapacity at peak, retries, degraded mode, pinned models and the drill that proves a fallback works.

Shock one, as a curve: where each design runs out of tokens per minute Illustration

Peak uncached input tokens per minute grow in proportion to traffic. B crosses the limit at x2.00, the prototype already at x0.67; C would need x6.7 (off the chart).

cost / USD 5 cap
peak tokens / limit at x2.5 (dashed: today)

Design C as its escalation share moves. Cost hits the cap at 62%; at x2.5 traffic the token limit binds at 80%. Quality rises with the share, so the experiment looks for the lowest share that clears 0.92 inside this window.

Arithmetic of the application block: hypothetical prices and limit, 90% cache hits on the 600-token prefix, cached reads assumed not to count against the limit. Quality is not on these axes; it comes from reviewed answers.

Cached reads are assumed not to count against the limit, as one provider documents for most models (as read 2026-10-11). Confirm with yours.

What the reliability gate contains Overview

Retries and deadlines
One retry owner, a deadline per request, idempotent writes (Day 52).
Degraded mode
Retrieval-only passages with citations, on a path that does not share the failing provider.
Pinned model
Dated version, a retirement calendar, the gate re-run well before the date.
Drilled rollback
One alias change away, and exercised with the failure injected.

A fallback only counts after a drill Diagram

Inject failureblock the provider
measure it
Fallback passes gates?
- yes: to Record the drill
no
Find shared dependencyauth, region, any hosted model call
- remove it: to Give it its own path
Record the drilldate, p95, error rate
Give it its own pathno shared failure domain
- re-run: back to Inject failure

Using cost headroom as an experiment2 minHow far C can escalate within the cap, and what that buys.

Bridging C from the small model to the cost cap (hypothetical prices) Bridge

Small model, every request0.7 USD/1k
30% escalated (today)0.30 × 6.94 = 2.78+2.08 USD/1k
30% → 45% (the experiment)3.82; 225k tok/min at 2.5x+1.04 USD/1k
45% → 62%the last headroom+1.18 USD/1k
Cost cap reached at 62%5 USD/1k

Cost = small model (0.70) + share x cached large model (6.94; B sends everything there). Escalation buys quality headroom with cost headroom. Cost binds before tokens per minute even at 2.5x traffic (capacity figure, right panel), so the search is for the lowest share that clears 0.92, on a review large enough to tell.

Negotiating with numbers, at staff leveladvanced3 minWhat one quality point is worth in the case's own currency, and what separates a staff-level defence from a typical one.

Negotiating with the case's own numbers (hypothetical prices) Table

Quantity	Value	How it is computed
Requests per month	1,200,000	40,000 a day x 30
One quality point	12,000 answers a month	0.01 x 1,200,000
B over the cost cap	USD 2,330 a month	(6.942 - 5.00) x 1,200
Gap per answer that one point would protect	about USD 0.19	2,330 / 12,000

If the business values a correct fee quote above about 0.19 per answer, one point of quality pays for B's overrun. The owner decides the value; the record states it.

What a staff-level defence adds Comparison

Criterion	Typical defence	Staff-level defence
Opens with	The chosen design and its strengths	The decision, its owners and the cost of a wrong answer
Treats a failed gate as	A reason to argue the gate	A specific experiment, or a negotiation with a named owner
Uncertainty	Point estimates, one table	Labelled evidence, interval or review size per gate
Ends with	A recommendation	What is given up and the numbers that would reverse it

The decision record, filled in3 minContext, decision, alternatives, labelled evidence, what was given up, triggers and owner.

The one-page decision record for this case Table

Section	Content
Context	120 agents quote tariffs and fees; a wrong fee costs more than the cost gap; launch in three weeks
Decision	Pilot B with 10% of agents behind a flag; retrieval-only passages as degraded mode
Alternatives	A (prototype), C (cascade 30%), retrieval-only; largest model on full context
Evidence	Simulated reviews (60 answers), hypothetical prices and limits; no live measurements yet
Given up	USD 2,330 a month over the cost cap, accepted by the budget owner
Triggers	C clears 0.92 within the cap; retirement notice; re-check capacity before traffic nears 2x today's peak
Owner and date	Named owner per trigger; review date in the record

B reaches the 400k limit at 2x today's peak (200k x 2), so the traffic trigger sits below that. Day 58 uses the same record and links it to an assumption register.

Memory anchor

Gates first, evidence labelled, trade-offs named, triggers as numbers.

Useful for

Preparing a design review, an interview system-design answer or a launch go/no-go for a GenAI feature, from a bank's dispute assistant to a SaaS onboarding copilot.
Turning an argument about models into a table of gates, evidence and owners.
Writing a decision record that a successor can re-evaluate when an assumption changes.

Not suitable for

Early prototyping, where gates and an attack suite would slow learning; use them before real users see the feature.
Choosing between designs without any labelled evidence; collect it first.

Traps

Moving the gate to fit the design

The favourite design passes, the gate no longer protects anyone and the record hides the compromise. A manufacturing team that lowers a defect-recall gate to fit its preferred model ships the misses to customers.

How to avoid it

Set gates with owners before results, and if a gate must change, record who changed it and why.

One table mixing measured and hypothetical numbers

Readers treat a hypothetical price like a measured latency, and the review approves a budget nobody has checked, as when a media company signs off a summarisation budget built on last year's list price.

How to avoid it

Label each number measured, simulated or hypothetical in the table and in the record.

Senior reasoning

A senior states the decision and its owners, fixes the gates, shows each design against them with labelled evidence, says which constraint is a negotiation (cost) and which is not (a wrong fee quoted to a customer), and ends with the numbers that would make them change their mind. They concede the valid part of every challenge.

Interview question

Design and defend a production RAG assistant over a hospital's clinical guidelines (a regulated domain) in ten minutes, including what would make you change the design.

Outline your answer first.

Recall cards scheduled after completion

What are the six steps of a gate-first design defence?
A candidate misses a gate by one reviewed answer out of 60. What do you do and not do?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

AI Engineering — Chip Huyen, 2024core
Applied-LLM text covering evaluation methodology, RAG and agents, and inference optimization; the integrated view this case exercises. (Chapters 3, 4, 6 and 9)
OWASP Top 10 for LLM Applications (2025 risk list) — OWASP Gen AI Security Projectcore
A shared risk checklist for the design review's security gate.
Model deprecations (Claude Platform Docs) — Anthropicoptional
Model lifecycle states and the advice to test replacement models well before a retirement date, which underlies the second shock.
Rate limits (Claude Platform Docs) — Anthropicoptional
Limits counted in tokens per minute with most cache reads excluded, which underlies the capacity gate; check your provider's current rules.
Canarying Releases (The Site Reliability Workbook, chapter 16) — Alec Warner, Stepan Davidovic et al., Googleoptional
Canary as a partial, time-limited rollout compared with a control, the basis of the flagged 10% pilot.
The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction — Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, D. Sculley, 2017optional
A production-readiness rubric of tests for data, model, infrastructure and monitoring, useful for the gate list.

Optional sections

Deep dive · 10 min

Why gates are set first. Thresholds chosen after results are a form of test-set reuse: the design under review quietly defines what counts as good. Owners disagree less about numbers they set before anyone has a favourite.

Which constraints negotiate. Cost and latency targets are budgets that can move with a business case. Correctness floors on critical slices, data-access rules and legal obligations do not move for convenience; if they are unmet, the right outcome is a narrower launch or a different design. Say which is which in the record.

Alternatives worth listing. Retrieval-only (show cited passages, no generated answer), a rules or SQL path for counting and aggregation questions, a smaller model with a validation step, a cascade, and the current prototype. A non-generative option clarifies what the generator adds and doubles as a degraded mode.

Security gate content. Name the untrusted inputs, the tools and their allow-lists, the identity used for each call, the approval points, the outbound content controls and the attack suite with its pass threshold and residual cases (Day 53). Include quotas and spend limits as controls, because cost is also an attack surface.

Reliability gate content. Provider error handling and retry ownership, a deadline per request, idempotency for writes, a fallback or degraded mode, a pinned model version with a retirement calendar, and a rollback that is one alias change away (Day 52).

The record. Context, decision, alternatives, evidence with labels, consequences including what was given up, revisit triggers with numbers, owner and date. Day 58 treats the format and its use in system design in more depth.

Interview practice · 10 min

At an insurer, a stakeholder says of the claims-policy assistant: "Your design is too complicated. Use the biggest model and put all the documents in the prompt." How do you respond?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-11

1 Recall · 5 min

2 Concept · 10 min