SQL, data modeling, data quality, EDA, visualization · insight-0046
Report business metrics by event time and pipeline health by processing time
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data analyst, ML engineer, Data scientist
- Industries
- e-commerce, telecom, aviation
Summary: situation, action, reason, boundary
Situation
A dashboard counts bookings by the timestamp at which records landed in the warehouse.
Action
Keep both timestamps: when the event happened and when it was ingested. Aggregate business KPIs by event time, monitor lag and completeness by processing time, and show the difference (ingestion delay) as its own metric.
Reason
Event time is fixed for each record, while processing time depends on system delays; the skew between them changes over time.
Boundary
Streaming events, mobile apps with offline mode, partner feeds and batch file drops.
Symptom or risk
A delayed batch makes Monday look empty and Tuesday look like a spike; backfills rewrite history unpredictably.
Exceptions
Operational metrics about the pipeline itself, such as throughput, are naturally in processing time.
Trade-offs
Event-time reporting means recent periods keep changing as late data arrives.
Pitfalls
Partitioning tables by load date and then filtering business queries on that partition.
Lessons and concepts
Claims and their sources
Event time does not change for an event, whereas processing time changes as it flows through the pipeline, producing a dynamically changing skew between the two.