S31 · stakeholder simulation
Readout memo for a fare-rules assistant pilot
People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.
- 4stages
- 35 minabout
- 0stages committed
The brief
The synthetic airline's customer care team piloted a retrieval-based assistant that answers fare-rules chat questions (baggage, seats and boarding, change and refund fees). All figures below are simulated for this exercise, and so is the head of customer care.
- Design: 8,000 chat sessions over four weeks, assigned at random to the assistant (4,000) or the existing FAQ bot (4,000).
- Primary metric: contained = no human agent contact within 48 hours. Assistant 66.5% (2,660 of 4,000), FAQ bot 61.0% (2,440 of 4,000). Difference +5.5 points, 95% interval +3.4 to +7.6.
- Answer audit (not part of the primary metric): care specialists checked 300 assistant answers and 300 FAQ-bot answers, 100 per topic in each arm, against the current fare rules. An answer is wrong if it contradicts the rules.
| Topic (share of sessions) | Assistant wrong | FAQ bot wrong |
|---|---|---|
| Baggage (40%) | 2 of 100 | 4 of 100 |
| Seats and boarding (30%) | 3 of 100 | 3 of 100 |
| Change and refund fees (30%) | 16 of 100 | 5 of 100 |
| All audited | 21 of 300 (7.0%) | 12 of 300 (4.0%) |
The head of customer care wants one slide and one page: "Do we roll this out to all chat traffic?"
Stage 1 of 4
The slide title is the sentence the head of customer care will repeat to colleagues.
Which slide title do you choose?