Engineering, production, governance, cost, communication · insight-0289

Declare model incidents early and give one person stakeholder communication

Situation
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
Action
Declare an incident, name an incident commander, an operations lead and a communication lead, keep a shared live incident document, and send updates on a fixed cadence stating impact, current action, what is unknown and the next update time.
Reason
Separating command, operations and communication roles and keeping a live document reduces confusion during response.
Boundary
Model outages, wrong predictions with customer impact, data pipeline failures.

Symptom or risk

Responders are interrupted, stakeholders hear conflicting stories, and nobody owns the decision to roll back.

Minor issues fixed within minutes may only need a ticket, but define the threshold beforehand.

Declaring early creates some false alarms; that is cheaper than late escalation.

Promising a fix time before the cause is known.

Google SRE incident management separates incident commander, operations and communication roles, keeps a live incident document and recommends declaring incidents early.
Managing Incidents (Site Reliability Engineering, chapter 14)