Engineering, production, governance, cost, communication · insight-1263

Test the trained artifact against the real serving binary, not just the training code path

Situation
Modeling code changes weekly, while the serving binary changes monthly, and a new model uses an operation or preprocessing step the deployed server does not know.
Action
In CI, build the candidate artifact, load it in the exact serving image, run end-to-end requests through the public interface and compare outputs with the offline path on a fixed sample before any traffic.
Reason
Mismatches between artifacts and serving infrastructure are a recurring failure that an integration test and a canary can catch.
Boundary
Teams where training and serving code have different release cycles or owners.

Symptom or risk

The model passes offline evaluation and then fails or misbehaves only when loaded by the server, usually at release time.

If training and serving are one repository and always released together, a single smoke test is enough.

Integration tests are slower and need maintained fixtures; run a small subset on each commit and the full run before release.

Mocking the model loader in tests, which removes the very interface that breaks.

The ML Test Score rubric lists integration testing of the full ML pipeline and canarying models before serving, and notes that mismatches between model artifacts and serving infrastructure are a recurring problem a canary can catch.
The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction