SQL, data modeling, data quality, EDA, visualization · insight-1059
Set the late-data allowance from the measured arrival-delay distribution
- Evidence type
- practitioner heuristic
- Difficulty
- hard
- Roles
- Data analyst, Data scientist, ML engineer
- Industries
- aviation, e-commerce, cross-industry
Summary: situation, action, reason, boundary
Situation
A streaming job aggregates bookings by event time with an arbitrary 10-minute allowance, and the daily total disagrees with the nightly batch by a stable fraction of a percent.
Action
Measure delay (processing time minus event time) per source over a few weeks, pick the allowance from a high percentile such as the 99th, report the share still dropped, and reconcile against a batch recompute. Route anything later to a correction path rather than discarding it.
Reason
A watermark bounds state by assuming how late data can be; the right bound is an empirical property of each source.
Boundary
Streaming aggregations, near-real-time dashboards and incremental loads with lookback windows.
Symptom or risk
Events that arrive after the allowance are dropped or never reach the window, so the shortfall concentrates in the sources that are slowest, for example mobile or partner channels.
Exceptions
Sources with guaranteed in-order delivery.
Trade-offs
A longer allowance increases latency and state size.
Pitfalls
Measuring delay only on events that were accepted, which hides the tail.
Lessons and concepts
Claims and their sources
In Spark Structured Streaming a watermark is the maximum event time seen minus a threshold, windows keep state until it passes, and data later than the threshold may be dropped, so the guarantee holds only for data within the threshold.
The Dataflow paper describes watermarks as heuristic lower bounds on event times, which can be too fast so late data arrives behind them.