Engineering, production, governance, cost, communication · insight-1271
Assign a prediction id at serving time and join delayed outcomes to it
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, insurance, e-commerce
Summary: situation, action, reason, boundary
Situation
Outcomes arrive weeks later keyed by customer and date, while the customer was scored several times by two model versions in the meantime.
Action
Generate a unique prediction id per decision, log it with model version, features reference and timestamp, carry it to the downstream action, and have the outcome pipeline join on that id with a documented fallback key.
Reason
Only an identifier created at decision time ties an outcome to the exact score and version that caused the action.
Boundary
Online and batch scoring where outcomes are delayed and entities are scored repeatedly.
Symptom or risk
Joining on entity and date attaches the outcome to the wrong score or to both, so per-version quality cannot be computed and replays disagree with live results.
Exceptions
Where each entity has exactly one decision in the outcome window, the entity key and decision date are enough.
Trade-offs
Needs the id to be propagated through partner systems, which is not always possible.
Pitfalls
Regenerating the id on retry, which turns one decision into two.