S01 · incident simulation

The load forecast that moved its peak by three hours

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
35 minabout
0stages committed

The brief

You are the on-call ML engineer for a regional grid operator (simulated company). A gradient-boosted model forecasts hourly load for the next day (24 values). Its strongest inputs are the weather vendor's hourly temperature forecast for the same hours, calendar features and recent load. The region runs on UTC+3 all year (no daylight-saving changes). The trading desk submits its day-ahead position at 11:00 local time each day (simulated deadline), using your forecast.

The model is retrained nightly on a rolling two-year window. All data, metrics and people in this scenario are simulated.

Stage 1 of 4

Thursday 2024-07-18, 07:40. An automated alert fires: forecast MAPE (mean absolute percentage error) over the last three days is 4.8% against a trailing 30-day level of 2.1%.

What you already know:

No code deploy since 2024-07-09; the model artefact is the nightly retrain, as usual.
Input checks are green: 24 rows per day for every feed, no nulls, temperatures inside the allowed range (-20 to 50 C).
The weather was hot but not unusual for July.

What do you do first?

Retrain the model now on the latest data so it adapts to the new conditions; the nightly retrain is clearly lagging whatever changed, and a fresh fit is the quickest way to find out.Slice recent errors by hour of day, compare each key input's daily profile with recent load, and check upstream feeds for changes (release notes, schema, timestamp format).Roll back to last week's model artefact, since the model is the only component that retrains by itself and a rollback is quick and reversible.Wait two more days; three days of high error in a heatwave can be noise, and acting on a short window risks chasing a transient.