Feature engineering and preprocessing · insight-0130
Normalize count and frequency encodings by window length so they survive changes in data volume
- Evidence type
- practitioner heuristic
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- e-commerce, saas
Summary: situation, action, reason, boundary
Situation
You encode a category by how often it appears, or an entity by its raw event count.
Action
Compute frequencies on the training fold, express them per fixed window (bookings per 30 days, share of rows), and recompute them on the same window at serving time.
Reason
Raw counts mix behaviour with data volume; rates over fixed windows isolate the behaviour the model should learn.
Boundary
Frequency encoding, activity counts and popularity features in growing products or after system migrations.
Symptom or risk
Counts grow with history length and traffic, so serving values drift away from training values even when behaviour is stable.
Exceptions
When total volume itself is the signal (a surge in route searches before a holiday), keep a separate volume feature.
Trade-offs
Short windows react quickly but are noisy; long windows are stable but slow to reflect change.
Pitfalls
Frequency maps computed on train plus test, which leak test-set composition into features.