Engineering, production, governance, cost, communication · insight-1260
Load the model artifact in a clean copy of the serving environment before release
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model is trained in a notebook environment with recent library versions and shipped as a pickle to a container with older ones.
Action
Pin the library versions used for training, save them and a data snapshot reference with the artifact, and in CI load the artifact in the serving image, score a fixed sample and compare to saved outputs.
Reason
Pickle-based formats are not supported across library versions, so compatibility has to be established by test, not assumed.
Boundary
scikit-learn, XGBoost and similar models persisted with pickle or joblib and deployed in a separate image.
Symptom or risk
The file may fail to load, load with warnings that nobody reads, or load and produce different results, and the failure shows up only after deployment.
Exceptions
Formats that run on a separate inference runtime still need a numerical-parity test but not a matching Python environment.
Trade-offs
Pinning slows library upgrades and forces periodic retraining against new versions.
Pitfalls
Treating a version warning as noise, and loading artifacts from untrusted sources because the format is convenient.
Lessons and concepts
Claims and their sources
The scikit-learn persistence guide says pickle-based models are not supported across scikit-learn or dependency versions, advises pinning dependencies and saving metadata with the model, and warns that pickle and joblib files should only be loaded from trusted sources.