Engineering, production, governance, cost, communication · insight-1275
After an incident, replay the affected window with the corrected model into a separate table
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, insurance, aviation
Summary: situation, action, reason, boundary
Situation
A feature bug corrupted scores for 38 hours, and decisions were made on them before the fix.
Action
Identify the window from logs, rescore the same inputs with the corrected pipeline into a separate table, diff against the decisions made, and decide per class of decision whether to reverse, compensate or accept.
Reason
A side-by-side diff quantifies the impact and gives each remediation an auditable basis.
Boundary
Batch and online decision systems with decision records and replayable inputs.
Symptom or risk
Without a bounded list of affected decisions, the team either ignores the damage or overwrites production tables with a rerun that cannot be audited.
Exceptions
Where decisions cannot be reversed or amended, document the impact and fix forward only.
Trade-offs
Replay needs retained inputs and costs compute; the retention policy limits how far back it can go.
Pitfalls
Replaying with today's features instead of the values available at decision time.