Engineering, production, governance, cost, communication · insight-1261
Make lineage queryable from dataset to models so a bad table has a known blast radius
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- banking, retail, cross-industry
Summary: situation, action, reason, boundary
Situation
A data engineer finds that last month's customer table had a bad join, and nobody can say which models trained on it.
Action
Record for every training run the dataset snapshots, code revision and feature definitions as run metadata, and keep the records in a store that can be queried by dataset to list models, deployments and downstream tables.
Reason
Lineage metadata modeled as jobs, runs and datasets lets you ask which runs consumed a dataset, which is the question an incident needs answered.
Boundary
Organizations with several models and shared feature or training tables.
Symptom or risk
The team either retrains everything at high cost or guesses, and some affected models stay in production for months.
Exceptions
A single model with one pipeline can use the run record in the experiment tracker.
Trade-offs
Capturing lineage requires instrumentation in every job and ongoing upkeep.
Pitfalls
Lineage that stops at the table level while the defect was in one column, or at the training job while the model is later fine-tuned elsewhere.
Lessons and concepts
Claims and their sources
The OpenLineage object model describes jobs, runs and datasets as core entities, where a job consumes or produces datasets and a run is one occurrence of a job.