Feature engineering and preprocessing · insight-0113
Add a missing indicator when the fact of missingness may carry signal
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- aviation, healthcare
Summary: situation, action, reason, boundary
Situation
A feature is missing more often for one segment or outcome, such as revenue missing more on intl-long flights in the synthetic raw table.
Action
Impute and add a binary missing flag (add_indicator=True) inside the pipeline, then check with cross-validation whether the flag earns its place.
Reason
When missingness depends on unobserved or outcome-related factors, the flag lets the model learn a separate effect for missing rows.
Boundary
Prediction tasks with MAR or MNAR missingness and stable collection processes.
Symptom or risk
Imputing the median erases the pattern; the model cannot distinguish a true median value from a missing one.
Exceptions
For causal or inferential estimates, indicator methods can bias coefficients; use multiple imputation or a missingness model instead.
Trade-offs
Flags add columns and can encode process artefacts that change when data collection is fixed.
Pitfalls
A flag that reflects a process timing (field filled only after the outcome) is leakage, not signal.
Lessons and concepts
Claims and their sources
MissingIndicator and add_indicator mark imputed positions because missingness can be informative.
Indicator-style approaches can give biased estimates in inferential analysis; multiple imputation is the principled alternative.