Engineering, production, governance, cost, communication · insight-1278draft — in review
Pin pipeline reads to the logical run date so a backfill reproduces history
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A feature job reads 'latest' from a source table or calls now() inside the task, and a three-week backfill is run after a bug fix.
Action
Parameterize each task by its data interval, read only that partition or an as-of snapshot, keep the task free of the current clock, and compare a backfilled day with the value originally produced.
Reason
A task that is a pure function of its logical date gives the same result on a rerun, which is what makes retries and backfills safe.
Boundary
Scheduled feature, label and scoring jobs built on orchestrators with backfill support.
Symptom or risk
Every backfilled day is computed from today's data, so features for past dates contain information from the future and the retraining set leaks.
Exceptions
Sources that keep no history cannot be pinned; snapshot them going forward.
Trade-offs
Needs retained partitions or time-travel snapshots, which cost storage.
Pitfalls
Pinning the main input but reading a mutable dimension table with current values.
Lessons and concepts
Claims and their sources
The Airflow best-practices page says tasks should give the same outcome when re-run, advises against using now() for critical computation inside a task, and recommends reading a fixed partition by data interval rather than the latest data.