Engineering, production, governance, cost, communication · insight-0257
Log served features and replay them to measure skew, not just predictions
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- banking, e-commerce, saas
Summary: situation, action, reason, boundary
Situation
Offline AUC was 0.82, and live performance on matured labels is clearly lower.
Action
Log a sample of request features, model version and prediction at serving time; recompute the same rows with the training pipeline and compare feature by feature; then rescore the logged features offline.
Reason
Replaying logged inputs separates input differences from model differences and points to the feature that diverged.
Boundary
Online and batch models where offline and online performance disagree.
Symptom or risk
Without the exact features the model received, you cannot tell whether the gap is skew, drift or a label definition change, so debugging becomes guesswork.
Exceptions
Where features contain sensitive data, log hashed, bucketed or sampled values under the privacy policy.
Trade-offs
Logging adds storage and privacy review; sampling keeps cost manageable.
Pitfalls
Logging only the post-imputation feature vector, which hides that the raw value was missing.