Engineering, production, governance, cost, communication · insight-0290
Run blameless postmortems with triggers defined before incidents happen
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer, Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
Action
Define postmortem triggers in advance (customer-visible impact, rollback, data loss, monitoring failure), write up timeline, contributing causes, detection gaps and owned action items, and add a test or monitor that would have caught the issue.
Reason
Blameless postmortems focus on contributing causes and preventive actions, and predefined triggers make writing them routine rather than political.
Boundary
Model, data and pipeline incidents, including near misses.
Symptom or risk
Blame makes people hide mistakes; without a written analysis the same weak point fails again.
Exceptions
Repeated identical incidents can share one postmortem with an updated action list.
Trade-offs
Postmortems take hours; skipping them costs more when failures repeat.
Pitfalls
Action items without owners or due dates.
Lessons and concepts
Claims and their sources
Google SRE postmortem culture is blameless, focuses on contributing causes, and defines postmortem triggers such as user-visible downtime, data loss, rollbacks and monitoring failures in advance.