S18 · lifecycle simulation

A procedures assistant from intake to retirement

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
50 minabout
0stages committed

The brief

Ramp agents at the synthetic airline ask questions about ground-handling procedures (loading limits, towing restrictions, de-icing intervals). The procedures live in 2,400 pages that are revised often. An intranet keyword search exists and is slow to use on the ramp. You are asked to build a retrieval-based assistant that answers with a cited page. All results, volumes and prices below are simulated or hypothetical.

Stage 1 of 5

The sponsor (simulated) says: "Success is that agents like it." On a labelled set of 200 questions (versioned as golden set v1) the existing keyword search returns the gold page in its top 5 for 61% of questions.

How do you define success and ownership at intake?

Track a thumbs-up rate from agents and aim for 80%; it is the measure closest to what the sponsor wants, it needs no labelled data to start, and it moves immediately when something goes wrong.Match a reference answer text for every question using an overlap score, because that is the usual automatic metric for generated answers and it can be computed on every release without reviewers.Beat the keyword search by launch day on any metric the team prefers, and refine the success definition once the system runs and the agents' real questions are better understood.Success is cited, faithful answers on golden set v1 with correct abstention, better than keyword search at finding the page, plus named owners for corpus freshness and evaluation.