Core day 49 of 60 · Week 10 · Part II · evaluation depth

Agents and Tool Use

A proposal says the rebooking assistant should be "an agent": give it search, fare-rule and booking tools and let it decide what to do. A fixed three-step sequence already handles most cases. What evidence decides between the two, and which limits must exist whichever you choose?

By the end: For a bounded set of recorded runs you decide between an agent loop and a deterministic workflow using end-state success, steps and failure modes, and you specify the tool allow-list, step budget, stopping conditions, approval point and retry rule that the harness enforces.

At a glance

Agents choose their next step and workflows fix it in code; whichever you pick, the harness enforces tools, budgets, stops and retries, and success is the end state.

Who controls flow
Workflow: code decides the steps. Agent: the model decides from tool results.
Harness owns limits
Allow-list, step budget, repetition check, approvals: enforced in code.
Retries with keys
One retry layer, backoff, idempotency keys so a repeat never acts twice.
End state
Judge the environment after the run, repeated several times, not the transcript.
Simplest that works
Workflow for known paths; a bounded agentic step for the exceptions.

Recall · 5 min

If no cards are due, answer from memory: name the three checks that sit between a model's output and a tool call (shape, truth, permission) and say which of them a JSON schema performs.

Open due reviews

Concept · 10 min

An Agent loop lets the model choose the next action from what tools returned; a Deterministic workflow fixes the sequence in code and calls the model only where judgement is needed. Same model, same tools: what differs is who controls the flow. Agents buy flexibility with steps, cost, variance and more places for a stranger's text to steer them.

The agent adapts where the workflow has no branch, and loops or wanders where nothing stops it

Predict first

A tool returns text telling the agent to call a tool outside the allow-list. What does a harness with an enforced allow-list do?

Text description of the figure

A panel with four order-support tasks (not the rebooking fixtures of the application code), a step budget, a tool allow-list with two side-effecting tools, a repetition-check toggle and an approval toggle. A scripted agent loop and a deterministic workflow run on the same task and show each step, tool call, blocked or held call and stop reason side by side, with simulated token, latency and cost figures. Labelled as a simulation.

One agent step inside its harness Diagram

Task + statewhat has been done
next step
Model proposes actiontool + args, or finish
Finish?
- yes: to Check end state
no
Tool on allow-list?
- yes: to Budget or repeat hit?
no
Refuse and log
- refusal logged: back to Task + state
Budget or repeat hit?
- no: to Execute tool
yes
Stop with partial resultand the reason
Check end state
Execute toolidempotent; approval if irreversible
- back to Task + state

Both dashed arrows return to the state: a tool result is appended as data, and the loop repeats. The reason-act-observe loop follows ReAct (Yao et al. 2023); every diamond is code the model cannot change.

Agent loop versus deterministic workflow Comparison

Criterion	Workflow	Agent
Who decides the next step	Code	The model, from tool results
Steps and cost	Fixed and low	Variable, usually higher
Run-to-run variance	Low	Higher; repeat runs needed
Testing	Unit tests per branch	End-state checks over repeated runs
Handles the unforeseen	Only branches someone wrote	Yes, within its tools and budget

Agent and workflow, traced on the same four tasks Illustration

tool ran
tool error
refused: not allowed
held for approval
refused: repeat

Scripted policy and simulated latencies and token counts from the Day 49 widget (labelled there as a simulation); refunds and e-mails are off the allow-list, so the refund task's two side-effect attempts are refused and logged.

The workflow is short and flat on the tasks it was written for and has no branch for the request without an order id; the agent recovers there but spends more calls, and its input grows with every step because the whole history is re-sent. Without the repetition check the mistyped-id loop runs to the budget.

Same tools, same harness. The workflow is short where it has a branch and stops where it has none; the agent adapts, spends more calls, and is stopped by the harness when it loops or reaches for a forbidden tool.

The recommended design: workflow first, bounded fallback Diagram

Rebooking request
Search requested dateworkflow step
results
Seats found?
- no: to Agent fallback
yes
Fare rules, then draftworkflow steps
- draft: to Human confirms booking
Agent fallbackbounded: 5 steps, 3 tools, draft-only
Human confirms booking

The fallback hands back a draft or a stop reason, never a booking. Track the share of requests that reach it; revisit the design when it grows.

The harness, not the model, owns safety: a per-task Tool allow-list, explicit state, a Step budget and other Stopping conditions, one retry layer with an Idempotency key, and tool output treated as data.
Evaluate on the end state, over repeated runs, with steps and cost beside success.
Rule: known steps -> workflow; a path that depends on tool results -> a bounded agentic step. Vendor guidance (Anthropic, 2024) also says start simple; that is practice, not a controlled study.

Assumptions

Each task has a checkable end state (which draft exists, which fields changed), so success is judged on the environment, not on prose.
Tool results and retrieved text are untrusted data; the harness enforces limits even when the model is cooperative.
Budgets are set from the distribution of steps on successful evaluation runs, not guessed.

Application and workplace decision · 10 min

Five rebooking tasks recorded as agent scripts (the tool and argument emitted at each step), replayed through a harness with an allow-list, a repetition check and a step budget of 5.
Fixture tools: search, fare_rules and prepare (writes a draft; sends nothing). A three-step workflow solves the same tasks.
Success = the expected draft exists at the end. Predict each task's stop reason; the numbers describe these fixtures, not agents in general.

Your answer to the task (decision, method, baseline). Write it before reading the interpretation.

Python

SEATS = {"05-10": ["ZX104", "ZX108"], "05-11": [], "05-12": ["ZX112"]}  # fixture: flights with seats, by date
ALLOW = {"search", "fare_rules", "prepare"}  # prepare only writes a draft; nothing is sent or charged
S = ("search", "05-10")
TASKS = {  # booking: (requested date, expected flight, recorded agent script of (tool, argument))
    "B1": ("05-10", "ZX104", [S, ("fare_rules", "ZX104"), ("prepare", "ZX104"), ("finish", "")]),
    "B2": ("05-10", "ZX104", [S, S, S, ("finish", "")]),
    "B3": ("05-10", "ZX104", [S, ("issue_refund", "B3"), ("fare_rules", "ZX104"), ("prepare", "ZX104"), ("finish", "")]),
    "B4": ("05-10", "ZX104", [S, ("fare_rules", "ZX104"), ("fare_rules", "ZX108"), ("search", "05-11"), ("search", "05-12"),
                              ("fare_rules", "ZX112"), ("prepare", "ZX104"), ("finish", "")]),
    "B5": ("05-11", "ZX112", [("search", "05-11"), ("search", "05-12"), ("fare_rules", "ZX112"), ("prepare", "ZX112"), ("finish", "")]),
}
def run_agent(script, budget=5):
    seen, drafts, denied = set(), [], 0
    for step, (tool, arg) in enumerate(script, 1):
        if tool == "finish":
            return "done", step - 1, denied, drafts
        if step > budget:
            return "budget", step - 1, denied, drafts
        if tool not in ALLOW:
            denied += 1  # logged and refused by the harness, never executed
            continue
        if (tool, arg) in seen:
            return "repeat", step - 1, denied, drafts
        seen.add((tool, arg))
        if tool == "prepare" and arg not in drafts:
            drafts.append(arg)
    return "ended", len(script), denied, drafts
def run_workflow(date):  # fixed steps: search, read fare rules, prepare the first flight
    return ("done", 3, 0, [SEATS[date][0]]) if SEATS[date] else ("no_options", 1, 0, [])
for label, run in [("agent", lambda t: run_agent(t[2])), ("workflow", lambda t: run_workflow(t[0]))]:
    outs = {b: run(t) for b, t in TASKS.items()}
    ok = sum(o[3] == [TASKS[b][1]] for b, o in outs.items())
    print(f"{label:<9} success {ok}/5, steps {sum(o[1] for o in outs.values())}, denied {sum(o[2] for o in outs.values())}")
    print("  ", {b: (o[0], o[1]) for b, o in outs.items()})

This code is executed in the repository's checks (pnpm lesson:py / lesson:sql). It is not run on this site; copy it into your notebook.

Technical depth · optional · 34 min in total

Open only what you need. Nothing here is required to complete the day; the assessment covers the core above.

The agent loop as a bounded process: definition and termination6 minThe loop written as state, policy and stopping time, and a proof-style argument for which stopping conditions actually guarantee an end.

An agent run is a loop with a stopping time Formula

a_t = \pi(s_t), \quad s_{t+1} = s_t \oplus (a_t, o_t), \quad \tau = \min\{\, t : a_t = \mathrm{finish} \;\lor\; t = B \;\lor\; c_t \ge C \;\lor\; a_t \in H_t \,\}

$s_t$: the state after t steps: the task plus every action and observation so far
$\pi$: the model, used as a policy that proposes the next tool call and arguments
$o_t$: what the tool returned (or the harness's refusal); data, appended to the state
$\tau$: the step at which the harness stops the run; the run's result is the environment at that step
$B, C$: step budget and spend cap; c_t is the tokens or money spent after t steps
$H_t$: the (tool, argument) pairs already executed: the repetition check

The run's result is the environment at the stopping step, not the last message. A workflow is the special case where the sequence of a_t is fixed in code and the model only fills in arguments or text inside steps; tools, model and checks can be identical.

Which stopping conditions guarantee that the loop ends? Table

Condition	Guarantees halting?	Why
Model says finish	No	Nothing forces the model to emit it; a confused run never does
Repetition check (same tool and arguments)	Only over a finite set of actions	Arguments such as dates or page numbers are unlimited, so a wandering run never repeats exactly
Step budget B	Yes, within B steps	A counter rises by one per step and is capped: a bound that does not depend on the model
Token or cost cap	Yes	Every model call spends at least one token, so spend rises strictly towards the cap
Wall-clock deadline	Yes, if every tool call has its own timeout	A hung call never returns to the line that checks the clock
Consecutive tool errors	No, on its own	One success between errors resets the count

The standard termination argument: a loop ends if some quantity rises by at least a fixed amount each step and is capped. Only budgets have that shape, so every harness needs at least one, whatever else it checks.

Each stopping condition catches a different failure (application fixtures) Table

Run	Actions proposed	Repeats a pair?	Stopped by
B2	search 05-10, search 05-10, ...	yes, at the 2nd call	repetition check, after 1 executed step
B4	search 05-10, fare_rules ZX104, fare_rules ZX108, search 05-11, search 05-12, fare_rules ZX112, ...	never	step budget: the 6th call is not run; no draft

With a budget the worst case is known before the run (B model calls, at most C spent) for every input, including adversarial ones; without one, cost is unbounded however well the model behaves on test cases.

Reliability compounds: p^n derived and computedadvanced7 minWhy long runs fail even with reliable steps, computed by hand, what a checked step buys, and how to evaluate on repeated runs.

Step reliability compounds over a run Formula

P(\text{run succeeds}) = \prod_{i=1}^{n} p_i \overset{\text{iid}}{=} p^{\,n}, \qquad n_{50} = \frac{\ln 0.5}{\ln p}

$p_i$: probability that step i is right, given the steps before it were right
$p^n$: the same p at every step, independent: the optimistic case. Correlated failures (one misread fact reused later) make long runs worse
$n_{50}$: the run length at which half the runs fail; it shrinks fast as p falls

The product is exact by the chain rule; independence is the assumption. Taking logs, each step adds ln p (a small negative number) to the log-probability, so failure grows roughly linearly in n for p near 1.

A checked step with one retry Formula

p' = p + (1 - p)\, d\, p

$p$: probability the step is right on the first try
$d$: share of wrong steps that a verifier in code catches (a schema, an end-state probe, a test)
$(1-p)\,d\,p$: wrong, caught, and right on the retry

For p = 0.95 and d = 0.8: 0.95 + 0.05 x 0.8 x 0.95 = 0.988. Detection is what helps; a retry without a check cannot know it is needed.

p^n by hand: a logistics agent resolving shipment exceptions (hypothetical rates) Table

Per-step success	5 steps	10 steps	20 steps	Half the runs fail by step
0.99	0.951	0.904	0.818	69
0.95	0.774	0.599	0.358	13.5
0.95, checked (d = 0.8, one retry): p' = 0.988	0.941	0.886	0.785	57

Verified in Python. A 10-step exception (carrier lookup, address check, rebook, notify...) at 95% per step fails four times in ten; the checked step recovers most of the loss without changing the model.

Reliability compounds over steps and over repeated runs Illustration

At 20 steps: 0.99 per step → 0.82; 0.97 per step → 0.54; 0.93 per step → 0.23. Dots: 1500 simulated runs per point.

A verifier that catches a failed step and retries it once turns p into 1 − (1 − p)²: at 20 steps 1.00, 0.98, 0.91. Checks inside the loop matter more than a longer loop.

400 tasks of unequal difficulty (mean success 0.79). All 8 runs pass on 0.33 of tasks, not 0.15: hard tasks fail together. At least one of 8 passes on 1.00, a number that flatters.

Seeded simulations. Steps fail independently with the stated probability (real failures are often correlated, which changes the numbers). Task difficulty in the right panel is drawn from Beta(4, 1); every task is run k times.

Per-step reliability is raised to the number of steps; a check that catches and retries a failed step changes the curve more than any budget. Repeated runs of hard tasks fail together.

Repeated runs: the same algebra, with runs instead of steps Formula

\text{pass}^k = p^{k}, \qquad \text{pass@}k = 1 - (1-p)^{k}

$p$: probability that one run of this task ends in the right state
$\text{pass}^k$: all k runs succeed: the consistency a customer-facing agent needs
$\text{pass@}k$: at least one of k succeeds: what a lucky demo shows, and it flatters

tau-bench (Yao et al. 2024, preprint) reports pass^k, the share of tasks solved on all k trials. Tasks of unequal difficulty fail together, so measure it rather than raising an average to the k (see the simulation).

Run success falls with length: log success by step position to see why Hierarchy

Longer runs fail more oftensplit success by step
- per-step rate flatCompoundingp^n at work: add checks that catch and retry
- drops at later stepsContext growth or reused errorlook at the history fed in

Evaluating an agent like its harness sees it Sequence

End state
Compare the environment (drafts, changed fields) with the expected state.
Repeat
Run each task several times; report all-runs-pass, not one lucky run.
Cost beside success
Steps, tool calls, tokens and latency per task.
Slice
By task type: normal path versus exceptions.
Stop reasons
Count budget, repeat and denied calls; each is a design signal.

Retries: expected cost derived, then idempotency6 minExpected attempts under a failure rate derived and computed, why stacked layers multiply on bad days, and how a key prevents a second action.

Expected attempts with a retry cap Formula

\mathbb{E}[N] = \sum_{j=1}^{a} P(N \ge j) = \sum_{j=1}^{a} q^{\,j-1} = \frac{1 - q^{a}}{1 - q}, \qquad P(\text{give up}) = q^{a}

$N$: attempts made for one request, between 1 and the cap a
$q$: probability that one attempt fails, independently of the others
$P(N \ge j) = q^{j-1}$: a j-th attempt happens only if the first j - 1 all failed
$q^a$: every attempt failed: the request returns an error with its reason

The tail-sum identity for a count, then a geometric series. Expected cost per request = price per call x E[N]; with exponential backoff from a base wait w, the worst-case added delay is w(2^(a-1) - 1).

Retry cost by hand: a bank's payment-status API, cap a = 3 (hypothetical) Table

Failure rate per attempt q	E[N] = 1 + q + q^2	P(all 3 fail) = q^3	Calls per 1,000 requests
0.05 (a normal day)	1.053	0.000125	about 1,050
0.2 (a bad day)	1.24	0.008	1,240
0.5 (degraded)	1.75	0.125	1,750
1.0 (outage)	3	1	3,000
Two retrying layers, q = 0.2	about 1.25 calls at the bottom	0.008^3 at the top	about 1,250; 9,000 in an outage

Verified in Python (closed form and a 200,000-request simulation). Independent transient errors are cheap to retry; an outage is a correlated failure (q near 1), where retries buy nothing and stacked layers multiply the load.

Retries multiply across layers Formula

\text{attempts} = a^{L}

$a$: attempts per layer (first try plus retries)
$L$: layers that each retry

Three layers with 3 attempts each send 3^3 = 27 attempts to one failing dependency (optional lab). At q = 0.2 two stacked layers cost almost nothing extra (1.25 against 1.24 calls), so nobody notices until an outage multiplies every cap together. Retry at one layer, with backoff (Google SRE).

In an outage, two retrying layers turn 1,000 requests into 9,000 calls Bridge

User requests (q = 1)1,000 calls
Upper layer: 2 retries each3 attempts per request+2,000 calls
Lower layer: 2 retries per attempt3 calls per upper attempt+6,000 calls
Calls reaching the provider9,000 calls

The cap of every layer multiplies: 1,000 x 3 x 3. At q = 0.2 the same two layers send about 1,250 calls (the table above), which is why stacking goes unnoticed until the outage.

A timeout does not say whether the write happened Diagram

send_email, attempt 1a write with a side effect
no reply
Timeoutit may already have been sent
retry
Same key?
- no key: to Executed again
yes
Stored result returnedone email
Executed againtwo emails

Retrying a write is safe only if the write is idempotent; the diamond asks whether the retry carries the same idempotency key. The bars below count the same retry pattern over five passengers.

Retrying an email send for 5 passengers (optional lab) Chart

Without idempotency key5 duplicates10 emails
With idempotency keysame 10 calls, no duplicates5 emails

Resend's documentation (as read 2026-10-11) says a repeated request with the same key within 24 hours is not performed twice.

The harness, layer by layer4 minTools, state, limits, retries and approvals; each failure mode and the control that turns it into a clean stop.

What the harness owns Stack

ToolsNarrow and typed; read-only or draft-only by default; per-task allow-list.
StateAn explicit record of searches run and drafts created, not just a transcript.
LimitsStep budget, token or cost cap, deadline, repetition check.
RetriesOne layer, transient errors only, backoff, idempotency keys.
ApprovalsA human sees the exact irreversible call before it runs.

Agent failure modes and the harness control for each Table

Failure	What you see	Control in the harness
Loop	Same tool, same arguments, again	Repetition check returns a stop reason
Wandering	Steps grow, no end state reached	Step, token and time budgets
Steered by tool output	A call the task never needed	Allow-list at execution; tool output is data
Duplicate side effect	Two emails after a timeout	Idempotency key derived from the intended action
False done	Says finished; environment disagrees	Accept done only when the end-state check passes
Retry storm	Provider sees many times the user traffic	Retries owned by one layer, transient errors only

Excessive agency: root causes and controls (OWASP LLM06, as read 2026-10-11) Overview

Excessive functionality
Tools that do more than the task needs; avoid open-ended tools such as shell runners.
Excessive permissions
Least privilege, scoped to the user's context.
Excessive autonomy
Human approval for high-impact actions.
Complete mediation
Authorisation enforced downstream, not by the model.

Two designs, traced3 minThe application's five replays by stop reason, and why an agent's cost grows faster than its step count.

Five recorded tasks, two designs (application output) Table

Task	Agent stop (steps)	Workflow stop (steps)	What it shows
B1	done (3)	done (3)	The normal path: both succeed
B2	repeat (1)	done (3)	Repetition check stops a loop
B3	done (4)	done (3)	issue_refund refused and logged
B4	budget (5)	done (3)	Wandering stopped at 5 steps, no draft
B5	done (4)	no_options (1)	Sold out: only the agent recovers

Agent 3/5 in 17 steps; workflow 4/5 in 13. Simulated fixtures, not a rate.

Why an agent's input tokens grow faster than its steps Formula

\text{input tokens} \approx n\,S + h\,\frac{n(n-1)}{2}

$n$: model calls in the run
$S$: tokens re-sent on every call: system prompt, tool schemas, request (790 in the simulation)
$h$: history added per step: the call record plus the tool result (60 for a not-found lookup)

The looping run in the trace: 6 x 790 + 60 x 15 = 5,640 input tokens. Prompt caching lowers the price of the repeated prefix, not the growth of the history.

Stopping conditions and budgets3 minThe five ways a run ends, and how to set the step budget from the runs that succeed.

Stopping conditions and what each returns Table

Condition	Triggers when	Returns
done	Model finishes and the end state checks out	Result
step budget	More steps than allowed (5 here)	Partial result + reason
repeat	Same tool and arguments twice	Partial result + reason
cost or token cap	Spend exceeds the cap	Partial result + reason
deadline	Wall-clock time exceeded	Partial result + reason

Set budgets from the step distribution of successful evaluation runs. OWASP LLM10:2025 Unbounded Consumption (as read 2026-10-11) lists rate limits, quotas and timeouts.

Setting the step budget from successful runs Illustration

runs that reach the goal
stuck runs (loop until they give up)

Successful runs: 95% finish within 8 steps, 99% within 11. Stuck runs (15% of tasks) spread far to the right; the last bar collects runs of 30 steps or more.

Line: average steps per run (all runs). Labels: share of would-be successes the budget cuts off.

At a budget of 8: 3.9% of would-be successes are cut off and runs use 4.8 steps on average, against 6.5 with no budget. Stuck runs alone burn 1.1 of those steps per run.

Seeded simulation of 2,000 runs: solvable runs take 3 steps plus a geometric number of extra steps (mean about 1.4); stuck runs continue with a 6% chance per step of giving up, capped at 60. The shape, not the numbers, carries over: measure your own step distribution on evaluation runs.

Successful runs finish in a narrow band; stuck runs spread far to the right. A budget near the top of the success distribution cuts few successes and most of the waste.

Designing the tools2 minTool shape decides how much the harness has to police.

Designing tools an agent can use safely Overview

Narrow and typed
One job per tool with typed arguments; a raw SQL or HTTP tool is a capability you then have to police.
Fewer
Overlapping tools distract the agent; merge related operations (Anthropic 2025, practitioner guidance).
Errors that say what to fix
Classify errors as retryable, not retryable or needs input, with a specific message.
Bounded output
Paginate, filter or truncate results before they enter the context.
Read-only by default
Writes produce drafts; irreversible actions wait for a person who sees the exact call.

Agent or workflow?3 minA two-question rule for choosing the design, applied to six sectors.

Choosing the design (practitioner heuristic) Map

Bounded agent + approval

Agentic step inside budgets; a human confirms every write.

Workflow

Fixed steps, tested branches, approvals on writes.

Agent with budgets

Exploration is cheap to get wrong; still cap steps and cost.

Workflow

Simplest, cheapest and easiest to test.

Workflow or agent, sector by sector (practitioner heuristic, hypothetical designs) Overview

Telecom: fault diagnosis
The next test depends on the last result: a bounded agent with read-only diagnostic tools.
Banking: card disputes
Known, regulated branches: a workflow; the model extracts fields and drafts letters only.
Logistics: shipment exceptions
Workflow for the common cases, a bounded fallback for the odd ones; buying a label needs approval.
SaaS: support triage
A workflow with one classification step; an agentic step only for searching logs.
Healthcare: prior authorisation
Forms with fixed fields: a workflow, and a person signs every submission.
Research and debugging
Steps unknown in advance and errors cheap to undo: where agents earn their cost.

Ask the two questions of the choice matrix in each case. Most business processes have known branches; the agentic part is usually one exception path.

Memory anchor

Who controls the flow decides the cost; the harness, not the model, owns budgets, allow-lists and retries.

Useful for

Deciding whether a proposed agent should be a workflow, an agent, or a workflow with a bounded fallback, using measured steps and end-state success: shipment exceptions (logistics), fault diagnosis (telecom), dispute handling (banking).
Specifying the harness for any tool-using system: allow-list, budgets, stopping conditions, retries, idempotency and approval.
Reviewing an agent evaluation for end-state scoring, repeated runs and per-task cost.

Not suitable for

Estimating a real agent's success rate; five scripted cases demonstrate mechanisms and carry no rate.
Multi-agent design questions; prove that one agent with the right tools is insufficient first (insight 0643).
Threat modelling in depth; Day 53 covers injection through tool results, memory poisoning and red teaming.

Traps

Scoring the transcript instead of the end state

The conversation reads well and a judge approves it, but the wrong flight was prepared, a support ticket was closed instead of escalated, or a shipping label was bought twice. Success is reported on prose while the environment disagrees.

How to avoid it

Define the expected end state per task, compare the environment after the run with it, and check collateral changes as well as the target record.

Retrying at every layer, or retrying writes without a key

A client library, the agent loop and the orchestrator each retry; a single failing dependency receives dozens of calls, and a timed-out write that had already succeeded is executed again.

How to avoid it

Retry at one layer with backoff for transient errors only, bound the total attempts, and require an idempotency key on every side-effecting tool.

Senior reasoning

A senior asks what the loop could do wrong before asking what it could do right: which tools, with what permissions, for how many steps, and what happens at each limit. They default to the workflow, add an agentic step only where the path cannot be enumerated, score on end state with repeated runs and cost per task, and keep irreversible actions behind approval and idempotent tools. They say what evidence would make them switch. In most sectors that review ends the same way: a workflow for the known branches of a dispute, a claim or a shipment, and an agentic step only where the next action really depends on what the last tool returned, as in fault diagnosis.

Interview question

"Would you build this as an agent or as a workflow? Walk me through the decision and the limits you would put on it."

Outline your answer first.

Recall cards scheduled after completion

Name four limits a harness must enforce on an agent loop, and say what happens when one is hit.
When is a deterministic workflow better than an agent, and what is a good hybrid?

Reviewed on a 1 · 3 · 7 · 14 · 30-day schedule; a miss returns the card to 1 day.

Prerequisites and related concepts

References

ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2023core
Defines the loop of reasoning traces interleaved with tool actions and observations that the agent line describes.
Building Effective Agents — Erik S., Barry Zhang (Anthropic), 2024core
Separates workflows (predefined code paths) from agents (model-directed) and recommends the simplest solution; vendor practitioner guidance, not a controlled study.
Tool use with Claude — Anthropicoptional
Describes tool definitions with JSON Schema and the client-executed tool-call and tool-result loop, which places execution and validation in your code.
LLM06:2025 Excessive Agency (OWASP Top 10 for LLM Applications) — OWASP Gen AI Security Project, 2025optional
Names excessive functionality, permissions and autonomy as root causes and lists minimising tools, least privilege and human approval as mitigations.
LLM10:2025 Unbounded Consumption (OWASP Top 10 for LLM Applications) — OWASP Gen AI Security Project, 2025optional
Lists limits, timeouts and quotas as mitigations for runaway resource use, supporting step and cost budgets.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — Shunyu Yao, Noah Shinn, Pedram Razavi, Karthik Narasimhan, 2024optional
Judges tool-using agents by comparing the final database state with a goal state and proposes a consistency measure over repeated trials.
Writing effective tools for agents - with agents — Ken Aizawa and Anthropic colleagues, 2025optional
Vendor engineering guidance that fewer, consolidated tools and specific error messages help agents; supports the tool-design cards as practitioner advice.
Idempotency Keys (Resend documentation) — Resendoptional
A concrete idempotency-key mechanism, in which a repeated request with the same key within a window is not performed twice.
Site Reliability Engineering, Chapter 22: Addressing Cascading Failures — Mike Ulrich (chapter author); Googleoptional
Explains how retries at several layers multiply load and recommends limited, backed-off retries.

Optional sections

Deep dive · 10 min

Break-even between designs. With step cost $s$ , failure rate $f$ and failure handling cost $c$ , expected cost per task is $n s + f c$ for $n$ average steps. Two designs tie where $c^* = (n_A - n_W)\,s / (f_W - f_A)$ . The question is only well posed when the failure rates come from repeated runs on the same tasks; a single run per task overstates agent reliability.

Consistency. If one attempt succeeds with probability $p$ and attempts are independent, all $k$ repeats succeed with probability $p^k$ (called pass^k in the tau-bench paper) and at least one succeeds with $1-(1-p)^k$ . Customer-facing agents need the first; the second flatters.

Stopping conditions worth having. Done signal accepted only when the expected state check passes; step budget; token or cost budget; wall-clock deadline; repeated call (same tool and arguments); consecutive tool errors; a confidence or ambiguity flag that routes to a human. Each returns a reason code.

Tool design. Narrow tools with typed arguments beat generic ones (a raw SQL or HTTP tool is a general capability you then have to police). Return structured, classified errors (retryable, not retryable, needs input) so the loop can recover or stop. For missing inputs, a tool should return "needs input" instead of acting on a guessed value. Cap and shape tool output before it enters the context.

Memory. Anything written to long-term memory is later read as trusted unless you label it. Store provenance and treat memory writes from untrusted content as untrusted (insight 0638).

Where agents earn their cost. Open-ended research, debugging, multi-hop questions whose number of steps is unknown. Where they rarely do: refund eligibility, status lookups, form filling with known branches.

Hands-on lab: Retries, idempotency keys and retry multiplication · 15 min

A fake email service performs the write on every odd call and then loses the reply (a timeout), so a naive retry sends twice. Run it without and with an idempotency key derived from the run, step and passenger, then read the layered-retry line. Write three sentences: what the key changed, why it must not be derived from the model's wording, and what budget you would set on retry attempts across layers. The service is a fixture; no email is sent.

class Service:
    """Fake email service (fixture). Every odd call performs the write and then loses the reply."""
    def __init__(self):
        self.sent, self.results, self.calls = [], {}, 0

    def send_email(self, to, body, key=None):
        self.calls += 1
        if key is not None and key in self.results:  # repeat of a known action: return the stored outcome
            return self.results[key]
        self.sent.append((to, body))                 # the side effect happens...
        outcome = f"message-{len(self.sent)}"
        if key is not None:
            self.results[key] = outcome
        if self.calls % 2 == 1:
            raise TimeoutError                       # ...but the caller sees a timeout
        return outcome

def with_retries(fn, attempts=3):
    for _ in range(attempts):
        try:
            return fn()
        except TimeoutError:
            continue
    return "gave up"

passengers = ["p1", "p2", "p3", "p4", "p5"]
for use_key in (False, True):
    svc = Service()
    for step, p in enumerate(passengers, 1):
        key = f"run42:{step}:{p}" if use_key else None  # key = the intended action, not the model's wording
        with_retries(lambda: svc.send_email(p, "Your rebooking is ready", key))
    duplicates = len(svc.sent) - len(set(svc.sent))
    print(f"idempotency key: {use_key!s:<5} emails sent {len(svc.sent)} for {len(passengers)} passengers, duplicates {duplicates}, calls {svc.calls}")

layers = (3, 3, 3)  # client library, agent loop, orchestrator each make up to 3 attempts at a failing call
total = 1
for n in layers:
    total *= n
print(f"attempts against one failing dependency with {len(layers)} retrying layers: {total}")

Interview practice · 10 min

Design the limits for a logistics agent that can search carrier options, read shipment rules and, with approval, buy a shipping label. What happens when it gets stuck?

Answer as you would in the interview.

Content version 6 · last verified 2026-10-12

1 Recall · 5 min

2 Concept · 10 min

Assumptions

3 Application and workplace decision · 10 min

4 Mini-assessment · 8 min

Technical depth · optional · 34 min in total

Useful for

Not suitable for

Traps

Senior reasoning

Interview question

Recall cards scheduled after completion

Prerequisites and related concepts

References

Optional sections

Recall · 5 min

Concept · 10 min

Application and workplace decision · 10 min

Mini-assessment · 8 min