project-040 · LLM and GenAI · aviation
Tool-calling agent vs deterministic workflow for rebooking proposals after a cancellation
After a cancellation, agents need rebooking proposals: later flights on the same route with spare seats. Should the airline build this as a deterministic workflow (parse the ticket, query, rank, draft), as a tool-calling agent that decides its own steps, or as a hybrid (model extracts fields, deterministic code does the rest)? Which design gives correct, policy-compliant proposals at acceptable cost and latency, and where does the agent add value?
At a glance
Data
Academy synthetic airline database plus a seeded set of synthetic disruption tickets (synthetic) · flight_day: 20,824 rows, one per flight_no x date, 2022-01-01..2024-12-31, 11 routes with 1-3 flight numbers each (checked by query); planned task set of 30 structured and 40 free-text tickets
Details on dataBaseline
Pure deterministic workflow: a regular-expression and rule parser for the ticket, one SQL query for later flights on the same route within N days, ranking by date then…
Details on baselineValidation
All arms run on the same tickets. Structured and free-text families are reported separately. Each model-backed arm runs every ticket three times to measure variability;
Details on validationMetric
Task success by family and ambiguity tag; invalid tool-call rate; mean tool calls and steps; tokens per ticket; p50/p95 latency; variability across the three runs;
Details on metric
Success criterion
Decision rule set in advance: choose the agent only if it beats the hybrid on task success for free-text tickets by an amount whose paired interval excludes zero, with zero unapproved write attempts and no more than twice the hybrid's cost per ticket (hypothetical prices). Otherwise ship the hybrid or the pure workflow. On structured tickets the workflow is expected to be exact; any agent error there is reported as avoidable risk.
Context
Day 49 asks when an agent is better than a workflow. The honest answer depends on whether the task requires choosing steps at run time. Rebooking on a fixed rule is mostly deterministic; ambiguity lives in the request ("tomorrow morning", "two kids", missing flight number). This project separates the two by using two ticket families. Tools are read-only except one draft-only write that requires human approval, following Day 47's boundary rules.
Data
- Dataset
- Academy synthetic airline database plus a seeded set of synthetic disruption tickets (synthetic)
- Licence
- Academy-authored synthetic data; no third-party licence
- Size
- flight_day: 20,824 rows, one per flight_no x date, 2022-01-01..2024-12-31, 11 routes with 1-3 flight numbers each (checked by query); planned task set of 30 structured and 40 free-text tickets
- Verification
- executed
Access
python/artifacts/airline_sample.sqlite, built by python/capstone/airline_sim.py (synthetic, no network). Tickets are generated by a seeded learner script from flight_day rows: structured tickets (fields given) and free-text tickets with labelled ambiguity. Expected actions for free-text tickets are written by the learner. The database has no departure times or connections, so "next options" means later dates on the same route.
Grain
One task = one ticket (cancelled flight_no and date, passenger count, constraints). Tool results are rows of flight_day.
| Column | Meaning |
|---|---|
| flight_day.flight_no / date / route | Flight identity at the departure-date grain |
| flight_day.capacity / bookings_final / passengers | Used to compute spare seats as capacity minus passengers (a stated simplification) |
| ticket_text | Synthetic agent or passenger message; free-text tickets omit or blur fields on purpose |
| expected_action | propose (with expected option set), ask_clarification (with the missing field), or escalate |
| ambiguity_tag | missing_date, relative_date, missing_flight, party_size, conflicting_info, none |
EDA and data-quality rules
Confirm flight_day has one row per flight_no x date and that capacity is never below passengers (both held when checked). Spare seats are positive on every row of the sample, so 'no seats left' cases never occur naturally - create them with a labelled fixture override instead of assuming the code handles them.
Oracle check: for structured tickets, the expected option set is computed by an independent SQL query and spot-checked by hand on five tickets.
Free-text labels: each ticket has exactly one expected action; ambiguity tags are balanced so every tag has at least five tickets.
Baseline
Pure deterministic workflow: a regular-expression and rule parser for the ticket, one SQL query for later flights on the same route within N days, ranking by date then spare seats, and a draft proposal. On free-text tickets it asks for clarification whenever its parser cannot fill a required field. No model calls, zero token cost.
Features
Tools exposed to the model arms (JSON-schema inputs): get_flight(flight_no, date), list_flights(route, date_from, date_to), seats_available(flight_no, date), and draft_rebooking(proposal) which only writes to a draft table and returns "pending approval". Guardrails outside the model: argument validation, date-window limits, a maximum of 8 tool calls, a 60-second timeout, and refusal of any write that bypasses approval. Every call is traced.
Candidate methods
Hybrid: model extracts ticket fields to a schema, deterministic workflow does the rest
Uses the model only where language understanding is needed; behaviour after extraction is testable and repeatable.
Tool-calling agent with the four tools and a bounded loop
Can ask follow-up queries the workflow did not anticipate; costs more calls and is harder to test.
Validation design
All arms run on the same tickets. Structured and free-text families are reported separately. Each model-backed arm runs every ticket three times to measure variability; success requires the correct action and, for proposals, the correct option set. Development tickets (one third) are used for prompt and tool-description changes; the rest are scored once after freezing.
Leakage checks
Tool descriptions and prompts must not contain answers from evaluation tickets; the oracle query is not exposed as a tool. The learner writes free-text labels before running any arm.
Bias and representativeness
Synthetic tickets are cleaner and more uniform than real messages and are written in one language. Report the ambiguity-tag slices; do not generalise success rates to real traffic.
Robustness checks
Inject tool faults (timeout, empty result, malformed row) and check that each arm retries within limits or escalates; include tickets that ask the system to skip approval and verify the draft-only write still holds.
Technical metrics
Task success by family and ambiguity tag; invalid tool-call rate; mean tool calls and steps; tokens per ticket; p50/p95 latency; variability across the three runs; unapproved-write attempts (must be zero executed).
Business interpretation
Correct proposals per agent hour saved (labelled assumption), cost per 1,000 tickets under a hypothetical price table, and escalation rate - the work that still lands on a person.
Error slices
Ticket family: structured vs free-text.
Ambiguity tag: relative dates and missing flight numbers usually separate the arms.
Tool-fault tickets: recovery behaviour under timeouts and empty results.
Implementation steps
Build the tool layer over SQLite with argument validation, limits and a trace log; unit-test each tool and the approval gate.
Implement the deterministic workflow and the oracle; generate tickets with a seed and label free-text tickets.
Implement the evaluation harness (success rules, cost from token counts and a hypothetical price table, latency, variability).
With a configured model: implement the hybrid (schema-validated extraction) and the agent (bounded loop); tune only on development tickets.
Score all arms once on frozen tickets; write the comparison and the architecture decision record.
Reproduction
Learner-authored; for example uv run python -m rebook.make_tickets --seed 21, uv run python -m rebook.run --arm workflow and, when a model is configured through environment variables, --arm hybrid and --arm agent. Without a model, only the workflow arm, the tool layer, the tickets and the harness run, and the report states that the agent and hybrid comparisons were not run. Record model id, version and date for every model-backed run.
Limitations
Spare seats ignore overbooking policy, fare rules and connections; the database cannot represent them.
Task success on synthetic tickets says little about real passenger messages.
Agent behaviour depends on the model version; a later version can change the conclusion.
Production and monitoring
Whatever wins, keep the deterministic core as the system of record and the model at the edges. Trace every tool call with ticket id, arguments and result size; alert on loop-limit hits and rising escalation. Writes stay draft-only with human approval. Version prompts and tool schemas together and gate changes on this task set.
Interview questions
When is an agent the wrong design even if it scores well in a demo?
How do you make an agent's evaluation reproducible when the same ticket produces different tool sequences?
Where do you enforce 'no write without approval' - in the prompt, in the tool, or both? Why?
Postmortem and alternatives
A common outcome is that the hybrid matches the agent on most tickets at lower cost, and the remaining gap sits in a few ambiguity types that could be handled by a clarification step. Discuss whether those cases justify an agent or a better extraction schema.
Rubric3 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Fair comparison | Same tickets, success rules and limits for all arms; families reported separately; repeated runs. | Variability and cost are part of the decision, with intervals on success differences. |
| Tool safety | Read-only tools, draft-only write with approval, limits enforced outside the model and tested. | Fault-injection results show recovery or safe escalation for every fault type. |
| Decision record | ADR states the chosen design and the evidence. | States the ticket types that would flip the decision and how production monitoring would detect them. |
References5
- Building Effective Agents
Workflow-vs-agent distinction and the case for simple composable patterns (vendor guidance).
- ReAct: Synergizing Reasoning and Acting in Language Models
Reasoning-and-acting loop that the agent arm follows.
- Tool use with Claude
Tool definitions with JSON-schema inputs and the tool-call loop.
- JSON Schema Specification
Validating tool arguments and the extraction schema.
- OWASP Top 10 for LLM Applications (2025 risk list)
Excessive agency as a risk category for tool-using systems.