SQL, data modeling, data quality, EDA, visualization · insight-1075
Record run-level lineage so an incident can be traced to the job run that produced a bad table
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist, Data analyst
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A revenue table shows a spike, and the team cannot tell which of three overnight jobs, running against which input versions, wrote it.
Action
Emit an event per run with a run id, job name, start and end, status, and the datasets and partitions read and written (OpenLineage-style), store it centrally, and surface the run id on output tables. Use it to find downstream tables to quarantine.
Reason
A run is a specific occurrence of a job, so linking datasets to runs makes the path from symptom to cause a lookup.
Boundary
Orchestrated batch pipelines, dbt runs and model training jobs that produce shared artefacts.
Symptom or risk
Without run identifiers and input and output references, investigations rely on logs and memory, and the bad data may keep flowing while the cause is searched for.
Exceptions
Single-script notebooks with no downstream consumers.
Trade-offs
Instrumentation effort and event volume; partial coverage gives false confidence.
Pitfalls
Recording the job name but not the input partitions.
Lessons and concepts
Claims and their sources
OpenLineage defines a Run as one occurrence of a job identified by a client-generated id, emitting timestamped state-update events that can attach input and output datasets, with facets for extra per-run metadata.