Feature engineering and preprocessing · insight-1108
Fix the missing-indicator column list, because the default mode ties the schema to the training extract
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
MissingIndicator runs with its default features='missing-only', and a column that was complete in the training extract develops gaps in a later feed.
Action
Set features='all' or an explicit column list when the downstream model needs a stable schema, set error_on_new deliberately, and test the fitted transformer on a frame with unexpected NaNs.
Reason
The default derives its column set from the fit data, so output structure becomes a function of one extract rather than of the input contract.
Boundary
scikit-learn pipelines that go to production with retraining, or feed monitors that expect fixed column names.
Symptom or risk
The number and identity of indicator columns depend on which columns happened to have gaps at fit time, so retrains change the schema and transform can fail on new gaps.
Exceptions
One-off analyses where a schema that follows the data is acceptable.
Trade-offs
All-column indicators add constant columns that a downstream selector or penalty must handle.
Pitfalls
Silencing the new-gap error with error_on_new=False, which hides a source that has started dropping values.
Lessons and concepts
Claims and their sources
MissingIndicator's default features='missing-only' covers only features with missing values at fit time, and error_on_new raises at transform when a feature complete at fit has missing values.