Engineering, production, governance, cost, communication · insight-1276

Record time to detect and the detecting signal for every model incident

Situation
Three incidents in a quarter were each noticed by a business user, and the postmortems list different root causes.
Action
In each postmortem record when the fault began, when it was detected, which signal detected it, and who, then review the distribution each quarter and add a monitor for each class found by a human.
Reason
Detection delay is comparable across incidents with unrelated causes and points to monitoring gaps that preventing individual causes will not close.
Boundary
Teams with more than a few production models and a postmortem practice.

Symptom or risk

Fixes address causes one by one while the real gap, how late the team learns about problems, remains invisible.

With only one or two incidents a year, review each in depth instead.

Reconstructing the start time can be difficult and requires logs from before the detection.

Using time to resolve as the only measure, which rewards quick patches.

The SRE book chapter on postmortem culture describes blameless postmortems that focus on contributing causes and preventive action items, with triggers defined in advance.
Postmortem Culture: Learning from Failure (Site Reliability Engineering, chapter 15)