Feature engineering and preprocessing · insight-1133
Run a leave-one-feature-out audit on a schedule to remove features that no longer earn their dependency
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer, Data scientist
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A production model has accumulated hundreds of features over several releases, many added in bundles or kept after a replacement arrived.
Action
Each quarter, retrain without each feature or feature group on identical folds, remove those whose loss is within the fold noise, and record their upstream owner and cost.
Reason
An input that adds little incremental benefit still creates a data dependency; underutilized dependencies can be found by leave-one-feature-out evaluation.
Boundary
Long-lived tabular models with many sources and several teams.
Symptom or risk
Redundant inputs leave the model exposed to upstream changes, outages and cost, and when one is removed or changes meaning the model degrades although it added no value.
Exceptions
Features kept deliberately for fairness checks or regulatory documentation.
Trade-offs
Many refits, and correlated features mask each other, so test groups too.
Pitfalls
Removing features by a single lower score without the fold-to-fold spread.
Lessons and concepts
Claims and their sources
Sculley et al. describe underutilized data dependencies (legacy, bundled, epsilon and correlated features) and say they can be detected by exhaustive leave-one-feature-out evaluation.