Engineering, production, governance, cost, communication · insight-0256
Compute training and serving features from one shared definition
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
Training features are built in a SQL notebook, and the online service reimplements them in application code.
Action
Define each feature once (a shared library, a feature store definition or a single SQL view) and call that definition in both training and serving; add a test that scores a fixed sample through both paths and compares features.
Reason
Training-serving skew is a known production failure, and duplicating feature logic in two code paths is a typical way it arises.
Boundary
Online inference, batch scoring with a separate pipeline, models ported between languages.
Symptom or risk
Small differences in windows, time zones, null handling or category mapping make the served model see different inputs from the one evaluated, and accuracy drops without any error.
Exceptions
When serving latency forbids the training computation, keep two implementations but test them for equality on every release.
Trade-offs
A shared definition can constrain the serving stack or add a feature-store dependency.
Pitfalls
Comparing only schemas and types, which pass while semantics differ.
Lessons and concepts
Claims and their sources
Data validation for ML in TFX treats training/serving skew as a key challenge for production pipelines.