Feature engineering and preprocessing · insight-0116
Replace sentinel values like -999 before they reach linear or distance models
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- Data analyst, Data scientist, ML engineer
- Industries
- aviation, manufacturing
Summary: situation, action, reason, boundary
Situation
A source system encodes missing or not-applicable as -999, 9999 or 0.
Action
Convert sentinels to NaN at ingestion via the data contract, then impute or use native missing handling, adding an indicator where meaningful.
Reason
Sentinels are labels in disguise; numeric models cannot know that -999 means unknown rather than very small.
Boundary
Legacy systems, survey data, sensor feeds and joined reference tables.
Symptom or risk
Linear and distance models treat -999 as a real extreme value, bending slopes and distances; summary statistics shift silently.
Exceptions
Tree models can isolate a sentinel with one split, so damage is smaller there, though monitoring still miscounts missingness.
Trade-offs
Cleaning at ingestion fixes every consumer but needs agreement with the data producer.
Pitfalls
A sentinel 0 that collides with genuine zeros, such as zero cancellations; use a separate flag from the source if available.
Example
Hypothetical (the sample DB has no zero competitor_price): competitor_price stored as 0 when no competitor flies the route would make the route look like a price war.