S16 · llm failure simulation

An oversized voucher late in a long baggage-claim conversation

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

A baggage-claims assistant (a language model with two tools, lookup_claim and issue_voucher) works inside an 8,192-token context window. The trace below is a safe, simulated fixture: names, amounts and IDs are invented, and the orchestration library's truncation behaviour is a configuration of this fixture, not a general property.

Policy (in the system prompt): vouchers are capped at 50 USD per claim; larger compensation needs a human agent.

Trace at turn 9 of one conversation (token counts are logged by the orchestrator):

Prompt part	Tokens
System prompt with policy	1,150
Tool schemas (lookup_claim, issue_voucher)	1,300
Retrieved passages (4 x 640)	2,560
Conversation history (8 turns)	3,900
New user message	120
Total input	9,030
Reserved for output	1,024

Orchestrator log line: context_limit=8192 reserved_output=1024 truncated=left. Tool call emitted at turn 9: issue_voucher(claim_id="C-1042", amount=500). The tool succeeded.

Stage 1 of 4

The customer asked at turn 9 whether they could receive compensation for the delay and the missing bag. The assistant replied that a 500 voucher had been issued. The reviewer says the assistant "ignored its policy".

What is the most likely explanation of the trace?

The model hallucinated: language models sometimes invent amounts, and the cap is only a sentence in the prompt, so a more capable model would have stayed under it.A prompt injection in a retrieved passage told the assistant to issue a large voucher, and the model complied because the injected instruction was placed after the policy text in the prompt.The history is too long: eight turns of 3,900 tokens exceed any sensible limit, so the retrieved passages should be removed from the prompt until the history fits.The input exceeded the window (9,030 against 7,168 after reserving 1,024 for output) and 1,862 tokens were cut from the start: the whole policy prompt (1,150) and 712 tokens of tool schemas.