S01 · incident simulation

A renamed API field quietly sets competitor fares to zero

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

5stages
40 minabout
0stages committed

The brief

You own the fare-recommendation model for the domestic network of the synthetic airline used in this course. Every hour it scores upcoming departures and recommends a fare level (L, M or H). Its inputs include recent bookings, days to departure, calendar flags and competitor_price, the lowest competitor fare for the same day, fetched from a fare-shopping API. In offline analysis competitor_price was the second most important feature.

For domestic routes, analysts auto-accept recommendations unless they look unusual (simulated process). All people, numbers and outputs in this scenario are simulated.

Stage 1 of 5

Wednesday 2024-09-04, 09:30. A pricing analyst (simulated) messages you: "Since yesterday afternoon the model recommends fare level L on 38% of domestic departures. It is usually around 12%. Did you change something?"

You check: no model deploy since 2024-08-28; the hourly scoring job ran successfully every hour.

What is your first move?

Compare serving inputs since yesterday with the training reference, feature by feature (distribution, share of zeros, share of defaults), most important features first.Check whether competitors launched a sale yesterday; a market-wide price cut would push our recommendations down, so the model may simply be reacting correctly to real conditions.Pause auto-acceptance for domestic recommendations while you investigate, so analysts stop approving fares from possibly broken outputs.Retrain on the last week of data so the model reflects the current market.