S24 · adr simulation

Load-factor anomaly tickets, runbook or investigating agent

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

The synthetic airline's network analytics team receives about 600 anomaly tickets a month (hypothetical volume), each saying that a flight's load factor fell more than 8 points below its same-weekday average. An analyst must name the likely cause and pass it to the owner (schedule, pricing, data, or station operations).

Two prototypes were evaluated on 40 seeded incidents, faults injected into a copy of the synthetic airline database (28 of a known class, 12 novel combinations or classes). All results are simulated.

Runbook W: fixed ordered checks (capacity or schedule change, fare tier change, competitor capacity or price, holiday or event calendar, data completeness and duplicates); outputs the first cause that passes or "unexplained".
Agent A: an LLM with a read-only SQL tool on a replica, the schema notes and a 15-step budget; writes a cause and the queries that support it.

Stage 1 of 4

Measure	Runbook W	Agent A
Correct cause, known classes (n = 28)	26	24
Correct cause, novel incidents (n = 12)	1	7
A cause stated that is not true	3 (1 known, 2 novel)	6 (3 known, 3 novel)
"Unexplained" or no cause	10 (1 known, 9 novel)	3 (1 known, 2 novel)
Hypothetical cost per ticket	0.05 USD	0.90 USD
Median time per ticket	6 s	70 s
Runs that hit the step budget	0	7
Write actions attempted	0	0

Which statement does this table support?

Agent A is the better system: it finds 31 of 40 causes against 27, and the extra cost per ticket is small next to analyst time, so replace the runbook and retire the five fixed checks.Runbook W wins: it is correct on 26 of 28 known incidents and costs 0.05 USD, and incidents that need an agent are rare enough that analysts can handle the 'unexplained' cases manually at lower total cost.Neither dominates. W is fast, cheap and rarely wrong on known classes; A resolves more novel incidents but states a wrong cause more often, and 12 novel cases make that difference imprecise.The two are equally accurate once the unexplained and wrong counts are combined, since W leaves 13 of 40 unresolved or wrong and A leaves 9, a difference that is within noise.