Core day 52 of 60 · Week 11 · Part II · application depth

LLMOps

The assistant calls a tool that adds a note to a booking. Attempt one gets a 429, attempt two times out, attempt three succeeds, and the booking shows two notes. Which failures were safe to retry, which were not, and what must the trace contain to prove it?

By the end: For one simulated LLM request you classify each failure, say which attempts were safe to repeat and why, name the fields a trace needs to reconstruct what happened, and state the gate and rollback path for a prompt release.

At a glance

Classify each failure by the state it leaves, retry in one layer, protect every write with a key or a read-back, and release prompts like code.

Refused or unknown
A 429 did no work; a timeout may have. Only the first is always safe to retry.
One retry owner
Retries at several layers multiply: 3 retries at each of 3 layers allow up to 64 calls.
Key names the intent
Reuse one idempotency key across attempts of one action; a new key per attempt protects nothing.
Trace every attempt
Model, tokens, finish reason, attempt, release id and key on every span.
Release by slice
Version prompt, model, settings and index together; canary, gate on slices, roll back by alias.

Recall · 5 min

Without notes: what makes a tool call safe to repeat (Day 49), and what must an agent do first after a timeout on a call that writes something?

Open due reviews

Concept · 10 min

An LLM feature is a distributed system with a model in the middle. Every step can fail, and the useful question is not "did it fail?" but "what state is the world in, and is trying again safe?" A 429 was refused before any work. A timeout means you do not know: the write may have run and only the reply was lost.

A 429 is safe to retry, a timeout after a write is not, until you know what state the world is in

Predict first

A write call times out after the service already stored the record. A blind retry without a key leaves how many records?

Text description of the figure

A simulated request timeline with injectable failures (429, timeouts, 500, 400) and a selectable retry policy. Each attempt shows its status, backoff wait and whether the side effect ran, so you can see the same script produce one record or two depending on the policy.

One request, its stages, and where each one fails Diagram

Gateway + contextquota, cache, prompt
Model callstreams tokens
- to Validate
refused
429 / 5xx / 529nothing ran: back off, one layer
Validateschema, finish reason
- to Tool write
stream breaks
Error mid-streammark partial, count it, never cache
Tool writechanges the world
no reply
Timeoutmay have written: same key or read back

Reads and generation only cost tokens when repeated. The last stage changes the outside world, so it needs the most care.

Classify the signal by the state it leaves, then act Hierarchy

Error signalwhat state is the world in?
- refusedNothing ranbut retrying is not always useful
  - invalid400 invalid requestyour bug: fix it, never retry unchanged
  - rate limited429 + retry-afterwait at least retry-after, retry
  - budget spent429, no retry-afterspend cap: alert and degrade
- provider error5xx or 529often transient: back off, retry or fail over
- state unknownTimeout, lost replysame key, or read back first
- stream brokeError after 200mark partial, count as a failure

Four outcome classes: refused, failed, unknown, partial. Codes are from one provider's documentation (as read 2026-10-11); other vendors use other codes, but the classes carry over.

Is this attempt safe to repeat? Diagram

Refused before work?
- yes: to Back off, retry
no
Could it have written?
- no: to Retry freely
yes
Same key every attempt?
- yes: to Retry, same key
no
Read state backthen decide; no blind resend
Back off, retryhonour retry-after, cap
Retry freelya read: costs tokens only
Retry, same keyservice replays the result

A 400 never enters this chart: it is your bug, and repeating it changes nothing. A timeout is an unknown, not a failure: record it as its own outcome and resolve it with the same key or a read-back.

Retries at several layers multiply, and only when it hurts Illustration

Each outer attempt restarts every inner layer, so attempts multiply. With 2 retries per layer, three layers make 27 provider calls for one request.

With 3 retries per layer, amplification is invisible on a normal day and explodes as the provider nears a full outage, exactly when it can least absorb extra load.

Computed: exact expected attempts for nested layers, each retrying a fixed number of times, with provider failures independent across calls (a simplification: real outages are correlated, which makes the right-hand end more likely).

Attempts multiply across layers, they do not add. The worst case (1 + r) per layer, multiplied, is reached in a full outage, when every layer retries into the same limit.

Key terms: backoff with jitter, Retry amplification, Idempotency key, Trace and span, Regression gate.
Classify, then act: refused, failed, unknown or partial. The error tree below is the map.
Retries live in one layer, with jittered backoff, a cap, a deadline and respect for retry-after.
Writes need a key reused on every attempt of one intended action, or a read-back before resending.
One release = prompt, model id, decoding settings, retrieval parameters and index, gated on slices and rolled back by alias.

Assumptions

The downstream tool is idempotent by design or accepts a key; if neither holds, a read-back is the only safe retry.
Error semantics (codes, retry-after, limits, cache rules) are as your provider documents them today; they differ by vendor and change.
Traces store metadata by default; prompts and answers need consent, redaction and retention rules (Day 53).
The golden set covers the slices a release could break.

Application and workplace decision · 10 min

Replay the opening incident against a simulated provider: attempt 1 returns 429, attempt 2 times out after the write ran, attempt 3 returns 200.
Three policies on the same script: blind (retry everything), keyed (one idempotency key on every attempt), read-back (after a lost reply, read the state instead of resending).
No API is called; outcomes are scripted and waits are random draws.
Predict the notes each policy leaves, then run. The last line prints worst-case provider calls when three layers each allow three retries.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

import random

random.seed(7)                                   # simulated outcomes: no API call is made

class NoteService:                               # fake downstream service that stores booking notes
    def __init__(self):
        self.notes, self.seen = 0, set()

    def write(self, key):
        if key is not None and key in self.seen: # same idempotency key: replay, do not write again
            return
        self.notes += 1
        if key is not None:
            self.seen.add(key)

def run(policy, script=("429", "lost_reply", "200")):
    svc, waited = NoteService(), 0.0
    key = "req-7f3a/add-note" if policy == "keyed" else None
    for attempt, status in enumerate(script, start=1):
        if status != "429":                      # 429 is rejected before any work; the others ran the write
            svc.write(key)
        if status == "200" or (status == "lost_reply" and policy == "read-back"):
            break                                # read-back: state is read, not resent
        wait = random.uniform(0, min(8.0, 0.5 * 2 ** attempt))   # exponential backoff, full jitter
        waited += wait
        print(f"    attempt {attempt}: {status:10s} backoff {wait:.2f}s")
    return svc.notes, waited

for policy in ("blind", "keyed", "read-back"):
    print(f"policy {policy}:")
    notes, waited = run(policy)
    print(f"  -> notes written {notes}, time spent backing off {waited:.2f}s")
print("worst-case provider calls per user request:", (1 + 3) ** 3)

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 25 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The retry process as algebra: expected attempts under a failure rate5 minThe geometric-series derivation of expected attempts and success probability under a retry cap, worked by hand at three failure rates and verified by simulation.

Attempt i happens only if every earlier attempt failed Hierarchy

Attempt 1always made
- succeeds, 1−fDone after 1 call
- fails, fAttempt 2reached with probability f
  - succeeds, 1−fDone after 2 calls
  - fails, fAttempt 3 = cap mreached with probability f²
    - succeeds, 1−fDone after 3 calls
    - fails, fGive upprobability f³ = f^m

Drawn for m = 3 under independent, constant f. Summing the reach probabilities 1 + f + f² gives E[A]; the give-up leaf gives P(success) = 1 − f^m. The same tree fits any retried dependency: an OCR service for claims documents, a payment gateway or an LLM provider.

Expected attempts under a failure rate, derived Formula

\begin{aligned} \mathbb{E}[A] &= \sum_{i=1}^{m} \Pr(\text{attempt } i \text{ happens}) \\ &= \sum_{i=1}^{m} f^{\,i-1} = \frac{1 - f^{m}}{1 - f} \end{aligned}

$f$: probability one attempt fails with a retryable error (assumed constant and independent)
$m$: maximum attempts = 1 + retry cap; attempt i happens only if the first i - 1 all failed, hence f^(i-1)
$\sum f^{i-1}$: a finite geometric series; the closed form follows from (1 - f) x (1 + f + ... + f^(m-1)) = 1 - f^m
$1/(1-f)$: the uncapped limit as m grows: at f = 0.5 a retrying client averages 2 calls, at f = 0.9 it averages 10

The same series gives the success probability 1 - f^m. As f approaches 1 (a full outage) E[A] approaches m: every request spends its whole retry budget, which is exactly when the provider can least afford it. With L layers each capped at m attempts, the outage worst case multiplies to m^L.

Worked by hand: m = 4 attempts (3 retries) at three failure rates Table

Failure rate f	E[A] = (1 - f^4)/(1 - f)	P(success) = 1 - f^4	Calls per success = E[A]/P = 1/(1 - f)
0.2 (normal)	(1 - 0.0016)/0.8 = 1.25	0.998	1.25
0.9 (incident)	(1 - 0.656)/0.1 = 3.44	0.344	10
1.0 (outage)	4 (the cap; take the limit)	0	infinite: only the cap and the deadline stop it

Simulation (numpy default_rng(0), 200,000 requests): mean attempts 1.249 and 3.437, success 0.998 and 0.345. In the last column the f^m terms cancel: calls per success are 1/(1 - f) whatever the cap. At f = 0.9 every success costs ten provider calls, which is why a retry budget stops retrying long before the math does.

What a retry cap buys: much at low f, little in an incident Chart

f = 0.2 (normal)
f = 0.9 (incident)

Each point adds one retry (cap r = 0 to 5), from E[A] = (1 − f^m)/(1 − f) and 1 − f^m. At f = 0.2 one retry already gives 0.96 and three give 0.998; at f = 0.9 three retries buy 0.34 for 3.4x the calls. Past a measured f, failing fast and degrading beats retrying.

The fragile assumption: in an incident, f rises with retry volume Loop

Provider saturatedshared limit, every client hits itrejects ↓
f risesfailures correlate across clientstriggers ↓
Clients retryevery layer, often in syncadds calls ↓
More callsper request, toward m per layeradds load ↓
Load > capacitysame users, many more callssaturates it ↓
↺ back to 1 · Provider saturated

This feedback loop is the retry storm. The tree and table above assume independent attempts; the fan-out demo shows the multi-layer version, and jitter, retry budgets and deadlines are what break the loop.

Backoff, jitter and deadlines3 minThe wait formula, why it is random, and what stops the loop.

Exponential backoff with full jitter Formula

w_n \sim \mathrm{Uniform}\left(0,\ \min(c,\ b \cdot 2^{n})\right)

$w_n$: wait before retry n
$b$: base delay (0.5 s in the application code)
$c$: cap on the wait (8 s in the code), so waits stop growing
$\mathrm{Uniform}$: random draw: clients that failed together do not retry together

If the provider sends retry-after, wait at least that long: it is the provider saying when capacity returns. A per-request deadline stops the loop whatever the attempt count.

Upper bound of the wait per attempt (b = 0.5 s, cap = 8 s) Chart

The actual wait is a uniform draw between zero and this bound.

Why the wait is random: 1,000 clients retrying after one outage Illustration

No jitter: every client computes the same wait, so 1,000 retries land in the same instant, four times over.

Equal jitter: wait = half the bound plus a random half. Spikes become plateaus.

Full jitter: wait drawn uniformly from zero to the bound. The same 4,000 retries arrive as a low, spread load.

Simulated with the trace widget's retry code: 1,000 clients, 5 attempts, base 1 s, multiplier 2, cap 8 s, no retry-after, seeded draws. The total number of retries is identical in all three panels; only their timing differs.

Identical waits synchronise clients into spikes that can knock a recovering service over again. Jitter keeps the retry count and spreads the timing.

Retry budgets, deadlines, breakers and fallbacksadvanced3 minThe controls that keep a provider incident from becoming your own outage, and how each one is misused.

Four controls that stop retries from becoming the outage Comparison

Criterion	Retry budget	Deadline propagation	Circuit breaker	Fallback or degrade
What it limits	Retries as a share of requests, per process	Work done after the caller has given up	Calls to a dependency that keeps failing	User impact while the provider is unavailable
Trigger	Budget spent: fail fast instead of retrying	Remaining time passed down with the request	Failure rate over a window; probe before closing	Typed error from the retry owner
Misuse	Per request only: a fleet can still multiply load	Fixed inner timeouts longer than the outer deadline	Opens on 400s, which are your bugs, not overload	A fallback model never put through the release gate

Retry budgets and deadline propagation follow the SRE chapter on cascading failures; circuit breakers and fallbacks are common practitioner patterns. Measure amplification directly: provider calls per user request, per layer, on every dashboard.

Idempotency keys in practice3 minWhat the server does with a key, which actions need one, and what to do when no key exists.

What the service does with an idempotency key Diagram

Request + keykey names the intent
Key seen in window?
- yes: to Return stored result
no
Do the workstore key + result
- first result: to Respond
Return stored resultno second write
stored result
Respondsame answer either way

One documented implementation checks the last 24 hours (as read 2026-10-11). Many services also reject a reused key with a different body.

Which actions need a key Overview

Naturally idempotent
Set status to X, read an order or a claim, generate text. Repeating gives the same world.
Needs a key
Append a note, send an email, issue a refund, charge a card, create a ticket.
Key scope
Unique per intended action across users; identical on every retry of that action.
No key available
Read the state back after an unknown outcome before any resend.

Same failure script, three policies (application output) Table

Policy	Attempt 1: 429	Attempt 2: lost reply	Notes written
Blind retry	No work done	Write ran, resent anyway	2 (duplicate)
Keyed	No work done	Resent with the same key: replayed	1
Read-back	No work done	State read, nothing resent	1

What a trace must contain3 minSpans per stage, the fields an on-call engineer needs, and the privacy default.

One trace per request, one span per stage Stack

Tracerequest id, user scope, release id, total latency, outcome (ok, failed, unknown, partial)
Gateway spanquota decision, deadline, retry owner, attempt count
Retrieval spanindex version, k, passage ids (not text by default)
Model spanmodel id, input and output tokens, finish reason, time to first token, attempt
Tool spantool name, idempotency key, status, write confirmed or unknown

Store metadata by default. Prompt and answer text need consent, redaction and a retention rule (Day 53). Attribute names can follow the OpenTelemetry GenAI conventions, which are still changing (vendor facts below).

Releases, canaries and rollbacks4 minVersion the whole configuration, gate on slices, canary, and handle a quality drop with no release.

A prompt release, gated like code Diagram

Candidate releaseprompt, model, index
All slices above floor?
- yes: to Canary
no
Holdbaseline stays live
Canarysmall share, release id on traces
Canary holds?
- yes: to Promote
no
Roll backone alias change
Promotealias moves

The candidate first passes every slice floor on the golden set; the canary is then compared with the baseline on slices, retries and cost per request, with the release id on every trace.

Hands-on lab: the candidate wins overall and still fails a slice floor Chart

Overall baseline0.883 pass rate
Overall candidate0.942 pass rate
Refund baseline0.85 pass rate
Refund candidatefloor 0.88, agreed before the run: hold0.85 pass rate
General candidate0.975 pass rate
Baggage candidate0.9 pass rate

Synthetic golden-set draws (seed 11). A gate on the overall rate alone would have shipped it.

A slice floor in a small canary is close to a coin flip Illustration

regressed release (true pass rate 0.84) is rolled back
good release (true pass rate 0.95) is rolled back anyway

Exact binomial probabilities for hypothetical pass rates. A slice that gets 5% of a small canary's traffic may need days to reach a few hundred requests: size the canary per slice, not per total.

Rule: roll back if the canary's slice pass rate falls below 0.9. A release that truly passes 0.84 should be caught; one that passes 0.95 should not.

The probability that the floor rule rolls back a regressed release, and a good one, as the canary sees more requests from that slice.

Quality moved and nothing was released Sequence

Freeze the inputs
Re-run the frozen golden questions; inputs cannot explain a change on identical inputs.
Diff the traces
Model id, finish reasons, token counts and latency before and after the change.
Check what can move under you
Undated model aliases, provider defaults, index refresh jobs, upstream documents.
Pin and re-gate
Pin a dated model version, make it part of the release, re-run the slice gate.

Caching without serving the wrong answer2 minExact-match, semantic and provider prompt caches reuse different things and fail differently.

Three things called a cache Comparison

Criterion	Exact-match response cache	Semantic cache	Provider prompt cache
What it reuses	A stored answer for an identical normalised request, user scope and release id	An answer to a similar question	Processing of an identical prompt prefix
Saves	The whole model call	The whole model call	Input price and time to first token; output unchanged
Main risk	Stale answers after a release if the key omits the release id	Serves an answer written for someone else's parameters	Any change early in the prefix invalidates everything after it

Restrict semantic caching to static, non-personal content. Never cache a partial stream or an answer that failed validation. Prompt caching is not a response cache (Day 54).

Vendor facts, datedadvanced2 minThe documented provider behaviour this lesson leans on, with the date it was read.

Vendor facts behind this lesson, as read 2026-10-11 Table

Fact	Source in the registry
Typed errors 400, 429, 500, 504, 529; a request id on every response	Claude API errors docs
Official SDK retries transient errors twice by default, with backoff, honouring retry-after	Claude API errors docs
Errors can arrive after a streamed 200	Claude API errors docs
Spend-cap 429 carries no retry-after and keeps failing	Claude rate limits docs
At least 60 days' notice before a public model is retired; retired models fail	Claude model deprecations docs
Same idempotency key honoured within 24 hours (one email API)	Resend idempotency docs
GenAI span attributes (request model, token usage, finish reasons) have status Development; content attributes are opt-in	OpenTelemetry GenAI semantic conventions

Re-check before relying on them. One vendor is a documented example, not a recommendation; other providers differ.

Memory anchor

Know the state before you retry; one retry owner; the key names the intent.

Useful for

Deciding which errors to retry, back off on, fail over from or surface to a user, in a SaaS support assistant as much as in any service that writes.
Writing trace requirements that let an on-call engineer explain a duplicate action or a slow request.
Designing a release gate and a rollback path for prompt, model and retrieval changes.

Not suitable for

Judging answer quality; traces show what happened, a golden set shows whether it was good.
Offline batch jobs, which need job-level timeouts and restart logic instead of per-request deadlines.

Traps

A new idempotency key per attempt

Every retry looks like a new action, so the key adds a field to the request and prevents nothing.

How to avoid it

Generate the key once per intended action (for example run id plus step number) and reuse it on every attempt.

Treating the status code as the outcome

A 200 on a stream or a timeout on a write is recorded as success or failure although the real outcome is partial or unknown; an insurance team that records a timed-out claim payout as "failed" discovers the double payment from the customer.

How to avoid it

Record the final stream event and the confirmed side-effect state, and keep unknown as a distinct outcome.

Senior reasoning

A senior reads an incident by asking what state the world is in after each attempt. They keep exactly one layer in charge of retries, make every side effect safe to repeat or checkable afterwards, put the release id and the key in every trace, and refuse to ship a prompt change that passes on average but breaks a slice.

Interview question

Design the retry and rollback behaviour of a retail bank's customer-facing assistant that can reverse card fees.

Outline your answer first.

Recall cards scheduled after completion

Why is a 429 safe to retry but a timeout on a write is not, and what makes the second attempt safe?
What goes in one LLM trace, and why must retries sit in one layer?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

Claude API errors (Claude Platform Docs) — Anthropiccore
Documented example of typed errors (400, 429, 500, 504, 529), request ids, SDK retry defaults and errors that arrive after a streamed 200. (Error codes and streaming errors)
Site Reliability Engineering, Chapter 22: Addressing Cascading Failures — Mike Ulrich (chapter author); Googlecore
Randomised exponential backoff, retry budgets, multiplication of retries across layers and deadline propagation.
Rate limits (Claude Platform Docs) — Anthropicoptional
Rate limits in requests and tokens per minute, 429 with retry-after, and a spend-cap 429 without it.
Idempotency Keys (Resend documentation) — Resendoptional
One concrete idempotency-key implementation, with a repeated key honoured within a 24-hour window.
OpenTelemetry semantic conventions for generative AI (spans and metrics) — OpenTelemetryoptional
GenAI span and metric attributes for model, tokens and finish reason; status Development, so names may change; content attributes are opt-in.
Canarying Releases (The Site Reliability Workbook, chapter 16) — Alec Warner, Stepan Davidovic et al., Googleoptional
Canary as a partial, time-limited rollout compared with a control, with roll back when it looks bad.
Model deprecations (Claude Platform Docs) — Anthropicoptional
Model lifecycle states, retired models failing, and the advice to test replacements well before retirement.

Optional sections

Deep dive · 10 min

Idempotency in practice. A key identifies one intended action. The server stores the key with the result for a limited window (one documented implementation checks the last 24 hours) and returns the stored result for a repeated key. Keys must be unique per action across users and reused across retries; many designs also compare the request body and reject a key reused with a different one. Natural idempotency (set status to X) needs no key; "append a note" and "send an email" do.

Backoff. Full jitter draws the wait uniformly between zero and an exponentially growing cap, which spreads retries from many clients instead of synchronising them. Honour retry-after when present, and add a retry budget so that a share of failing requests cannot become a multiple of normal load.

Streaming. The status line is sent before the body. Decide what the user interface shows when an error arrives after text has started, and never write partial output to a cache or to a downstream system.

Caching. Prefer exact-match caching keyed on the normalised request, user scope and release id. A semantic cache that serves similar questions can return an answer written for someone else's parameters; restrict it to static, non-personal content. Provider prompt caching reuses an identical prefix and is not a response cache, so stable content goes first.

Releases and models. Treat prompt, model id, decoding settings, retrieval parameters and index as one versioned release; stamp its id on traces. Pin dated model versions and watch deprecation schedules. When a model is replaced, run the golden set on the candidate well before the retirement date.

Error-budget thinking. Pick a few signals (success rate, p95 latency, cost per request, refusal rate, retry rate) per release and alert on user-visible symptoms. Detail on latency decomposition is Day 54.

Hands-on lab: A slice-level regression gate for a prompt release · 15 min

The code draws synthetic golden-set outcomes for a baseline release and a candidate release across three slices, applies a gate (candidate must not fall below the baseline overall and each slice must stay at or above a floor), and prints the decision. Run it, then (1) explain why a gate on the overall rate alone would have shipped the candidate, (2) change the refund floor to 0.85, note what flips, and say why a floor chosen after seeing the result is a problem, and (3) add a rule that blocks a release when the candidate's canary retry rate exceeds the baseline's by more than 20%.

import numpy as np

rng = np.random.default_rng(11)
slices = {"general": 80, "refund": 20, "baggage": 20}          # synthetic golden set, 120 cases
base_p = {"general": 0.88, "refund": 0.90, "baggage": 0.85}    # pass probability, baseline release
cand_p = {"general": 0.95, "refund": 0.75, "baggage": 0.90}    # candidate: better on average, worse on refunds
floor = {"general": 0.80, "refund": 0.88, "baggage": 0.80}     # per-slice gates agreed before the run

def passes(p):
    return {s: int((rng.random(n) < p[s]).sum()) for s, n in slices.items()}

base, cand = passes(base_p), passes(cand_p)
total = sum(slices.values())
print("overall baseline", sum(base.values()) / total, "candidate", sum(cand.values()) / total)
ok = sum(cand.values()) >= sum(base.values())
for s, n in slices.items():
    rate = cand[s] / n
    flag = "ok" if rate >= floor[s] else "BELOW FLOOR"
    ok = ok and rate >= floor[s]
    print(f"{s:8s} baseline {base[s]}/{n} candidate {cand[s]}/{n} floor {floor[s]:.2f} {flag}")
print("gate decision:", "ship to canary" if ok else "hold; keep baseline release live")

Interview practice · 10 min

An e-commerce assistant's gateway shows a spike in duplicate orders after a provider incident. What do you look for and what do you change?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-13

1 Recall · 5 min

2 Concept · 10 min