Engineering, production, governance, cost, communication · insight-1266
Roll back the model together with its preprocessing, schema and stored state
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model v7 is rolled back to v6 after a regression, but v7 introduced a new feature column, a new encoder file and updated entries in an online store.
Action
Version the model, preprocessing code, encoders and feature schema as one release bundle, make schema changes backward and forward compatible for one release, and rehearse the rollback on a staging copy.
Reason
A rollback is a deployment of an older consumer into data produced by a newer producer, so it needs the compatibility that forward-compatible schema rules provide.
Boundary
Online stores with schemas that change between model versions, pipelines with stateful encoders.
Symptom or risk
The old model reads data written for the new schema, or the old code meets a changed encoder, so the rollback fails or produces a second incident.
Exceptions
Stateless models over unchanged inputs can roll back by changing a pointer.
Trade-offs
Compatibility windows delay cleanup of old fields and slow schema changes.
Pitfalls
Deleting the previous version's artifacts as part of the same release.
Lessons and concepts
Claims and their sources
Confluent schema compatibility types distinguish backward compatibility, where new consumers read old data, from forward compatibility, where old consumers read new data, and set the order in which producers and consumers must be upgraded.
The ML Test Score rubric lists the ability to roll back serving models as an infrastructure test.