S16 · llm failure simulation

A refund assistant that paid twice, in the wrong unit

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

A synthetic airline pilots an assistant that handles schedule-change refunds by calling two tools. This is a fixture: a fictional booking system, simulated model behaviour and no real payment provider.

lookup_booking(pnr)  -> {"fare_minor": int, "currency": str, "paid": bool, "refunds": [..]}
issue_refund(pnr, amount, currency, reason)
    amount: refund amount as a decimal number in the given currency

The harness retries a tool call when the model asks again; there is no idempotency key. lookup_booking returns the fare in minor units (cents). The refunds list in lookup_booking is updated by a separate job and lags by up to a minute.

Stage 1 of 4

Simulated trace for one request:

1 lookup_booking("QX7K2M")  -> {"fare_minor": 18450, "currency": "EUR", "paid": true, "refunds": []}
2 issue_refund("QX7K2M", 18450, "EUR", "schedule change") -> TIMEOUT after 10 s (no body)
3 issue_refund("QX7K2M", 18450, "EUR", "schedule change") -> {"refund_id": "R-90311", "status": "queued", "amount": 18450.0}
4 lookup_booking("QX7K2M")  -> {"fare_minor": 18450, "currency": "EUR", "paid": true, "refunds": []}
5 issue_refund("QX7K2M", 18450, "EUR", "schedule change") -> {"refund_id": "R-90312", "status": "queued", "amount": 18450.0}
final message: "I have refunded 184.50 EUR to the passenger."

The fixture ledger afterwards shows R-90311 and R-90312, each 18,450.00 EUR. Whether call 2 also executed before the timeout is unknown.

Which defects are in the system design rather than in the model's wording? Select all that apply.

The tools use different units for the same quantity, and the refund tool accepted an amount 100 times the fare without a bound check.The prompt did not contain a sentence reminding the model to be careful with money and to double-check every amount.A retry after an ambiguous timeout carried no idempotency key, so a repeat became a second refund instead of a no-op.The final message came from the model's belief, not a ledger read-back, so it stated an outcome nobody had verified.The lagging refunds list in lookup_booking proves the lookup tool is unreliable and should be removed from the agent.