SQL, data modeling, data quality, EDA, visualization · insight-0047

Mark recent periods as provisional until late-arriving data has had time to land

Situation
Yesterday's bookings look 15% low on the morning dashboard, and someone escalates a demand drop.
Action
Measure how much of each day's final volume typically arrives after 1, 2 and 7 days, and grey out or annotate periods that are not yet complete. Reprocess a trailing window on each run so late records update their original period.
Reason
Completeness estimates such as watermarks are heuristics and late data can still arrive after them, so recent aggregates are estimates until the arrival window has passed.
Boundary
Daily KPIs, monitoring metrics, label pipelines and training tables with recent cutoffs.

Symptom or risk

Incomplete recent data is read as a real decline, triggering price changes or incident calls.

Sources with guaranteed synchronous writes and no offline clients.

Waiting for completeness delays signals; showing provisional numbers with a label keeps speed without false alarms.

Training models on the most recent days before their labels and features are complete.

Watermarks are heuristic completeness estimates that can be too fast, so late data can arrive after them.
The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing
An incremental filter that reprocesses the most recent period lets records arriving later update existing rows when a unique key is set.
Configure incremental models (dbt Developer Hub)