Engineering, production, governance, cost, communication · insight-0299

Retire models whose lift over a simple rule no longer pays for their upkeep

Situation
A churn model built three years ago still runs, but recent evaluations show it barely beats a tenure-and-usage rule.
Action
Review each production model yearly against the current best simple alternative, estimate the value of its lift and its full running cost, and decommission or replace it when the balance is negative, keeping the rule as fallback.
Reason
Models are long-lived liabilities as well as assets; maintenance cost continues after the launch benefit fades.
Boundary
Portfolios of production models, especially older ones with unclear owners.

Symptom or risk

Every model carries pipelines, monitoring, on-call and review cost; marginal models consume capacity needed elsewhere.

Regulated models may need a formal replacement process before retirement.

Retirement work is unglamorous; make it part of the model lifecycle policy.

Comparing the old model with a rule on a period the rule's thresholds were tuned on.

Rules of ML guidance favours simple heuristics and simple first models when they meet the need.
Rules of Machine Learning: Best Practices for ML Engineering