Engineering, production, governance, cost, communication · insight-1259
Version reference tables and mappings as model inputs, and freeze them until the new version is vetted
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- retail, aviation, finance
Summary: situation, action, reason, boundary
Situation
A model uses a lookup table, such as airport-to-region mapping or product-category codes, that another team refreshes whenever it likes.
Action
Store each version of the table immutably, record the version id with the model, serve the version the model was trained with, and promote a new table only through the same evaluation as a model change.
Reason
Input signals that change behaviour over time, such as data-dependent lookups, make downstream models unstable; a frozen versioned copy removes the surprise at the cost of staleness.
Boundary
Category mappings, taxonomy tables, tax and fee schedules, outputs of other models used as features.
Symptom or risk
A refresh that looks like an improvement shifts the inputs under a model fitted to the old mapping, and the accuracy loss is hard to trace.
Exceptions
Where the mapping is defined by regulation and must always be current, version it and switch the model and table together.
Trade-offs
Several live versions have a maintenance cost and risk serving stale mappings.
Pitfalls
Versioning the model but reading 'latest' from the lookup table at serving time.
Lessons and concepts
Claims and their sources
Sculley et al. describe unstable data dependencies, including data-dependent lookup tables and inputs produced by other models, and name a versioned (frozen) copy of the signal as a common mitigation with its own costs such as staleness.