Skip to content
ML Academy

Engineering, production, governance, cost, communication · insight-0272

Define reproducibility as recoverable inputs plus tolerances, not identical bits

Evidence type
documented fact
Difficulty
medium
Roles
ML engineer, Data scientist
Industries
banking, insurance, healthcare

Summary: situation, action, reason, boundary

  1. Situation

    An auditor asks you to reproduce last year's model, and retraining gives slightly different numbers.

  2. Action

    Version the training data snapshot, code, dependency lockfile, config and seeds for every registered model, and define an acceptable tolerance for metric differences on re-runs.

  3. Reason

    Bit-identical results are not guaranteed across library releases, platforms or CPU and GPU, so reproducibility needs recorded inputs and stated tolerances.

  4. Boundary

    Regulated models, published analyses, any model you may need to explain later.

Symptom or risk

Without the data snapshot, code commit, environment and seeds, you cannot tell whether differences are noise or a real change.

Exceptions

Exploratory notebooks need lighter tracking, but anything promoted needs the full set.

Trade-offs

Deterministic algorithms can be slower; enable them only where audits need them.

Pitfalls

Pinning the code but querying a mutable 'latest' table for training data.

Lessons and concepts

day 19day 15
Reproducibility and versioningExperiment tracking and model registry

Claims and their sources