Skip to content
ML Academy

project-036 · LLM and GenAI · aviation

Flagship: passenger-rights document Q&A with a retrieval baseline, one controlled improvement and an honest generation mode

An airline customer-care team wants an internal assistant that answers agents' questions about passenger-rights rules (EU Regulation 261/2004, the Commission's 2024 interpretative guidelines, and the US DOT rules in 14 CFR parts 250, 259 and 260) by pointing to the governing paragraph, and that says "not covered by these documents" instead of guessing. Can a small retrieval system put the right paragraph in front of the agent often enough to justify a pilot, and which failure types would make a pilot unsafe?

Licence
EUR-Lex documents: reuse authorised under Commission Decision 2011/833/EU; eCFR: U.S. Government work (17 U.S.C. 105), check for embedded third-party material; query labels are your own work
Reference run
none

At a glance

  • Data

    Passenger-rights corpus: Regulation (EC) 261/2004, Commission interpretative guidelines C/2024/5687, 14 CFR parts 250/259/260, plus a learner-labelled query/evidence set (public) · Five documents; expect a few hundred paragraph-level passages (count and record after parsing); 60-80 learner-written queries

    Details on data

  • Baseline

    BM25 over paragraph passages with library-default parameters (k1 around 1.2, b around 0.75, recorded), lowercase tokenisation, no stemming, top-5 returned.

    Details on baseline

  • Validation

    Split the queries roughly 40/60 into dev and held-out, stratified by querytype, before any indexing.

    Details on validation

  • Metric

    Recall@5 and MRR@10 on answerable queries, nDCG@10 with graded relevance (grade 2 governing, 1 supporting), abstention precision and recall on unanswerable queries, all…

    Details on metric

Success criterion

Decision rules written before any held-out run (targets, not known to be achievable):

  • Pilot go: on the frozen held-out queries, the chosen retriever reaches recall@5 of at least 0.80 on answerable queries, and the improvement over BM25 is claimed only if the paired-bootstrap 95% interval of the recall@5 difference excludes zero.
  • Abstention: on unanswerable held-out queries the abstention rule abstains on at least 80% while abstaining on no more than 10% of answerable queries.
  • Safety: zero top-1 citations from the wrong jurisdiction on queries that name an EU or US itinerary; any such case blocks the pilot regardless of averages.
  • Generation mode (only with a configured model): at least 90% of cited passage ids are labelled relevant, judged on 30 sampled answers. A result that misses a target is still a valid project outcome if the report explains why and what would change the decision.

Context

This is the Weeks 9-12 learner flagship (spec section 7, Retrieval/RAG contract). Week 9 builds the corpus and the versioned query/evidence set (Day 43's rule: evaluation set before any adaptation). Week 10 builds the retrieval flow, schema/tool boundaries and one controlled improvement (Days 47-50). Week 11 runs the ablation, failure and security checks (Days 51-53). Week 12 writes the report and defends the design (Days 55-57). The domain is deliberately awkward: agents ask in everyday language ("four hours late from Frankfurt to Boston, what do we owe?") while the rules are drafted in legal language, the 2024 guidelines paraphrase the regulation at length, and EU and US rules answer similar questions differently. Everything here is educational; the system gives agents a paragraph to read, not legal advice. Project-037's reference implementation shows the retrieval-evaluation mechanics on a different corpus; it scaffolds this project and does not replace your own artefacts.

Data

Dataset
Passenger-rights corpus: Regulation (EC) 261/2004, Commission interpretative guidelines C/2024/5687, 14 CFR parts 250/259/260, plus a learner-labelled query/evidence set (public)
Licence
EUR-Lex documents: reuse authorised under Commission Decision 2011/833/EU; eCFR: U.S. Government work (17 U.S.C. 105), check for embedded third-party material; query labels are your own work
Size
Five documents; expect a few hundred paragraph-level passages (count and record after parsing); 60-80 learner-written queries
Verification
identity verified

Access

EUR-Lex HTML for CELEX 32004R0261 and OJ C/2024/5687 (no login). eCFR versioner API for title 14, parts 250, 259 and 260, pinned to one "as of" date (no login). Save the raw files once, with retrieval date, URL and SHA-256 in a manifest; evaluation never re-fetches. Fallback if automated access is blocked: download the HTML/PDF by hand and record that in the manifest. Last-resort fallback: run the retrieval-evaluation workflow on the academy corpus from project-037 and state that the flagship domain was not completed. The guidelines are registered as dataset-eurlex-261-guidelines-2024 and the CFR parts as dataset-ecfr-14-cfr-250-259-260.

Grain

Retrieval unit: one passage (one article paragraph or CFR paragraph) carrying document id, jurisdiction, version date and section path. Evaluation unit: one query with its set of relevant passage ids graded 2 (governing rule) or 1 (supporting context), or an explicit unanswerable flag.

ColumnMeaning
passage_idStable id such as eu261-art7-p1 or cfr259-5-b-2; never regenerated between runs
doc_id / jurisdiction / version_dateSource document, EU or US, and the version date recorded in the manifest
section_pathHeading hierarchy prepended to the passage when the contextual-header option is on
textPassage text after Unicode normalisation; footnotes handled by a recorded rule
query_id / query_textLearner-written question phrased the way an agent would ask it
relevantList of passage_id:grade pairs; empty list only when answerable is false
answerableFalse when the five documents do not govern the case (e.g. a rail itinerary, a non-EU non-US flight)
query_typelookup, paraphrase, numeric, exception, cross-jurisdiction, multi-passage, unanswerable, adversarial
splitdev (used for development and threshold tuning) or heldout (frozen, evaluated once)

EDA and data-quality rules

  • Parse completeness: every article/paragraph listed in each document's structure appears exactly once in the passage table; empty passages and passages above the chosen token limit are counted and resolved by a written rule.

  • One version per document: the 2016 guidelines are excluded (or tagged superseded and kept out of the index); the manifest records URL, retrieval date, as-of date and SHA-256 for each file.

  • Labels are written before you look at any retrieval output, from realistic agent scenarios; the held-out file is hashed and frozen before the first experiment, and the hash goes into the report.

  • Coverage minimums: at least 15 unanswerable queries, at least 10 cross-jurisdiction queries, at least 8 exception queries (extraordinary circumstances, oversales exemptions). Flag any query whose word overlap with its own evidence exceeds a Jaccard of 0.5 and rewrite it, otherwise BM25 is being tested on copy-paste.

  • Label audit: re-label 20% of queries after at least 48 hours (or ask a peer) and record disagreements; every relevant passage id must resolve to an existing passage.

Baseline

BM25 over paragraph passages with library-default parameters (k1 around 1.2, b around 0.75, recorded), lowercase tokenisation, no stemming, top-5 returned. Abstention: abstain when the top-1 BM25 score is below a threshold chosen on dev queries only. Report two floors next to it: "always answer" and "always abstain", so the abstention metric has a reference point. Do not tune the baseline harder than the candidate.

Features

Passage construction is the main feature decision: paragraph passages with the section path prepended (contextual header) versus bare paragraphs, chosen on dev. Metadata fields (jurisdiction, version date, document type) are stored for filtering and for the report's error slices. Query-side processing stays minimal: Unicode normalisation and a rule-based jurisdiction detector (EU airport or carrier, "DOT", US city pairs) that is itself evaluated on the labelled queries before it is allowed to filter anything. No query rewriting with an LLM in the baseline, because it would hide whether retrieval itself works.

Candidate methods

  • Hybrid BM25 + dense embeddings fused with reciprocal rank fusion

    Targets vocabulary mismatch between agent language and legal drafting. Needs a small open sentence-embedding model (a model download, CPU is enough at this corpus size). If downloads are not allowed, use LSA (TF-IDF + truncated SVD) as the dense stand-in and say so. RRF avoids calibrating BM25 and cosine scores against each other.

  • Metadata-aware retrieval: jurisdiction filter/boost plus contextual headers

    Choose this if dev failure analysis shows wrong-jurisdiction or wrong-document hits dominate. No model needed, cheap to explain to compliance, and its failure (a wrong jurisdiction guess) is visible and testable.

  • Cross-encoder reranking of the top 20 (optional, only after one of the above)

    Reorders near-duplicate passages between the regulation and the guidelines. Adds a model download and latency per query; measure both. Not a substitute for fixing first-stage recall, because it cannot rerank what was not retrieved.

Validation design

Split the queries roughly 40/60 into dev and held-out, stratified by query_type, before any indexing. All development, the choice of the single controlled improvement, the abstention threshold and the passage format are decided on dev. The held-out set is evaluated once per configuration in the final comparison; if you must run it again after a change, report both runs. Comparisons are paired on the same queries; report the paired bootstrap interval (1,000 resamples over queries, fixed seed). With 40 held-out queries the interval will be wide; that is a finding, not a reason to drop it.

Leakage checks

The person who writes the queries also tunes the retriever, so results are optimistic. Mitigations: write queries from scenarios before building the index, freeze and hash the held-out file, rewrite high-overlap queries, tune nothing on held-out. In generation mode, never put held-out evidence into few-shot examples, and record the model's training cut-off where the provider states it, because the regulation is public and may be memorised - an answer without a cited passage is not evidence of retrieval.

Bias and representativeness

Coverage is EU and US rules only, written by one author in one register of English. Report results by query_type and jurisdiction, and add ten queries in simplified or non-native phrasing to see whether lexical retrieval penalises them. Passengers whose case falls outside both regimes get "not covered", which is correct behaviour but must be routed to a human rather than read as "no rights".

Robustness checks

Typo and paraphrase perturbations on ten dev queries; a version-conflict test (index the 2016 guidelines in a scratch index and check whether stale text outranks the 2024 text); a prompt-injection test using one clearly labelled synthetic fixture passage, kept outside the official corpus, that contains an instruction to promise a large cash payment. Retrieval may surface it; generation mode must quote it as text, never follow it, and the report must say how the fixture was isolated from the production index.

Technical metrics

Recall@5 and MRR@10 on answerable queries, nDCG@10 with graded relevance (grade 2 governing, 1 supporting), abstention precision and recall on unanswerable queries, all by query_type. Generation mode adds citation precision (cited passage ids that are labelled relevant), a supported-claim rate from your own reading of 30 sampled answers, and the share of answers that correctly say "not covered".

Business interpretation

Share of agent questions where the governing paragraph is in the top five (the agent can verify it quickly), wrong-jurisdiction rate (the costly error: promising EU compensation on a US-only itinerary, or the reverse), and abstention rate (questions routed to a specialist). Any handling-time or cost translation is a labelled assumption, not a measurement.

Error slices

  • Query type: numeric thresholds (distance bands, delay hours) and exception queries usually fail differently from plain lookups.

  • Document: regulation vs guidelines - the guidelines restate articles at length, so the 'right' passage is often a near-duplicate of a supporting one.

  • Jurisdiction: EU-only, US-only and mixed itineraries; the mixed slice is where metadata filters help or hurt.

  • Answerable vs unanswerable: plot the top-1 score distributions of both groups on dev; overlap shows why a single threshold has limits.

Implementation steps

  • Fetch the five documents, write the manifest (URL, date, as-of date, SHA-256) and commit it with the raw files or a fetch script.

  • Parse into passages with stable ids; write unit tests for parse completeness and id stability; record passage-count and length statistics.

  • Write 60-80 queries with graded evidence before indexing; split dev/held-out by query type; hash and freeze held-out.

  • Implement the metrics module (recall@k, MRR, nDCG with graded labels, abstention precision/recall) and test it on a hand-computed toy case.

  • Run the BM25 baseline and the two floors on dev; classify at least 20 misses into failure types (vocabulary, jurisdiction, near-duplicate, exception, parse).

  • Pick one candidate that targets the dominant failure type, change only that, compare on dev, then run the final held-out comparison once.

  • Tune the abstention threshold on dev; evaluate it on held-out; run the robustness and injection checks.

  • Generation mode, only if a real model is configured (environment variable, no key in code): require answers to cite passage ids or say 'not covered'; log model id, version, date, prompt version and token usage; evaluate 30 answers by hand. Without a model, label the report 'retrieval-evaluation mode: generated-answer competence not assessed'.

  • Write the 500-800 word report, a one-page design defense, and complete the transfer task (a new document or question type you did not tune on).

Reproduction

Learner-authored repository (no academy code is provided for this project). Expected entry points, for example uv run python -m flagship.fetch --as-of 2026-10-07, uv run python -m flagship.index, uv run python -m flagship.evaluate --split dev and uv run python -m flagship.evaluate --split heldout --final. Pin dependency versions, fix seeds for bootstrap and any sampling, keep the manifest and the held-out hash in the repository, and record CPU time per step. Generation mode reads the provider key from the environment and writes a run log; reproducing it requires the same model version, which providers may retire.

Limitations

  • A 40-query held-out set gives wide intervals; small differences between configurations are not decision-grade.

  • Single-author labels: relevance judgements reflect one reading of legal text, and the audit only measures self-consistency or one peer.

  • Retrieval success is not a grounded answer: a correct passage in the top five says nothing about what a generator does with it.

  • Rules change: a new guideline version or CFR amendment silently stales the index unless the manifest date is checked.

  • Educational system: it ranks paragraphs for a trained agent; it does not determine entitlement.

Production and monitoring

Version the corpus by manifest date and re-index on a schedule with a diff report of changed passages. Log query, retrieved ids, scores, abstention decision and (in generation mode) model version and citations, with retention limits because agent questions may contain passenger data. Monitor abstention rate, wrong-jurisdiction flags raised by agents, and the share of answers without citations. Keep the held-out set as a regression gate for every index, chunking or prompt change. Retrieved text is untrusted input: tools are read-only and no answer can trigger a payment or a customer message without human approval. Rollback means re-pointing to the previous index version.

Interview questions

  • Your recall@5 improved from the baseline on dev but the held-out interval includes zero. What do you tell the product owner, and what would you do next?

  • How did you prevent your own query-writing from favouring BM25, and how would you know if it did?

  • Why is retrieval recall not enough evidence to launch an answer-generating assistant? What extra evaluation does generation require?

  • An agent reports the assistant cited an EU article for a US domestic flight. Walk through the diagnosis from logs to fix.

Postmortem and alternatives

Write the postmortem around the failure type that cost the most, not the average metric. Typical alternatives to discuss: a rules-first router (detect jurisdiction and question type, then search only the relevant document) may beat any embedding upgrade on this corpus; a smaller, better-labelled query set may have been worth more than a second candidate; and if generation mode never ran, say what evidence a reviewer is still missing.

Rubric5 criteria

CriterionMeetsExceeds
Evaluation set60+ queries with graded evidence, unanswerable and cross-jurisdiction coverage, frozen held-out with a recorded hash.Label audit with measured disagreement, overlap check applied, and a clear account of how labelling could bias results.
Controlled improvementOne change chosen from dev failure analysis, compared paired against BM25 with a bootstrap interval.Ablation isolates the effect (e.g. headers on/off within the hybrid), with cost and latency recorded per configuration.
Failure and security analysisError slices by query type and jurisdiction; injection fixture and version-conflict tests run and reported.Concrete mitigations tied to each failure type, with a test that would catch regression.
Honest generation claimGeneration mode either evaluated with a logged real model or explicitly labelled as not assessed.Citation precision and unsupported-answer behaviour analysed on sampled answers, separated from retrieval results.
Report, defense and transfer500-800 word report with a decision, limitations and the transfer task completed.Defense anticipates the strongest objection and states what evidence would reverse the decision.

References11