Engineering, production, governance, cost, communication · insight-0298
Look for hidden feedback loops when a model influences its own future training data
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- ML engineer, Data scientist
- Industries
- aviation, banking, e-commerce
Summary: situation, action, reason, boundary
Situation
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
Action
Map where predictions affect future labels or features, keep a small randomised or exploration slice of decisions, log the decision taken with each outcome, and evaluate on data from the exploration slice.
Reason
Hidden feedback loops are a known form of ML technical debt that makes system behaviour hard to analyse.
Boundary
Pricing, ranking, credit limits, fraud blocking, inventory allocation.
Symptom or risk
The model learns from outcomes it caused, so errors reinforce themselves and the data no longer show what would happen under other decisions.
Exceptions
Forecasts that do not change the forecast variable (weather) have no such loop.
Trade-offs
Exploration costs some short-term value; size it from the value of the information.
Pitfalls
Measuring a fraud model on transactions it allowed, ignoring those it blocked.
Lessons and concepts
Claims and their sources
Sculley and colleagues identify hidden feedback loops among ML-specific sources of technical debt.