Engineering, production, governance, cost, communication · insight-0274

Put data, feature and model-quality tests in CI, not only unit tests for code

Situation
The ML repository's CI runs Python unit tests, and a broken join still reached production.
Action
Add CI stages for schema and data expectations on a fixed sample, feature unit tests (known inputs to known outputs), a train-on-sample smoke test, and a quality gate comparing the candidate against the baseline on a frozen evaluation set.
Reason
Production-readiness rubrics for ML cover tests for data, models, infrastructure and monitoring, not only code.
Boundary
Any repository that trains or serves models.

Symptom or risk

ML systems also fail through data and model behaviour that raises no exception, so code-only CI misses a whole class of defects.

Full retraining in CI is often too slow; use small samples there and full checks in the release pipeline.

More gates slow merges; keep fast checks in CI and heavy ones nightly.

Quality gates on an evaluation set that changes silently between runs.

The ML Test Score is a rubric of tests for data, model development, infrastructure and monitoring for production readiness.
The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction