Feature engineering and preprocessing · insight-0126
Compute entity history features as of the prediction timestamp with point-in-time joins
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer, Data analyst
- Industries
- banking, aviation
Summary: situation, action, reason, boundary
Situation
You add aggregates such as a customer's booking count, a route's average load factor or an account's late payments to each training row.
Action
For every row, filter source events to timestamps strictly before the row's prediction time (minus any reporting delay), then aggregate; use a point-in-time join or feature store that does this by construction.
Reason
A feature must match what the serving system could have known at decision time, as it was recorded then.
Boundary
Churn, credit, fraud, demand and maintenance models built from event logs or slowly changing tables.
Symptom or risk
Aggregates computed over the whole table include events after the prediction time; offline scores are inflated and production scores drop.
Exceptions
Static attributes that never change (aircraft type of a fixed tail number) need no temporal filter.
Trade-offs
Point-in-time joins are slower and more complex than a single group-by, but the simple version measures the wrong thing.
Pitfalls
Using the latest snapshot of a dimension table that was overwritten (slowly changing dimension type 1) as if it held historical values.
Lessons and concepts
Claims and their sources
Point-in-time joins attach, for each entity and event timestamp, the latest feature values available at that time, preventing future values from leaking into training data.