Feature engineering and preprocessing · insight-1130
Keep only features that survive repeated subsampling when selection is noisy
- Evidence type
- documented fact
- Difficulty
- hard
- Roles
- Data scientist
- Industries
- healthcare, manufacturing
Summary: situation, action, reason, boundary
Situation
A lasso or importance-based selector returns a different shortlist when a few hundred rows change, and the team has to freeze a feature set.
Action
Run the selector on many random half-samples, record each feature's selection frequency, and keep features above a pre-set frequency threshold, reporting the frequency table.
Reason
Subsampling with a selection algorithm yields selection probabilities, and features that are chosen consistently are less likely to be sampling noise.
Boundary
High-dimensional tabular data, with lasso, elastic net and tree-importance selectors.
Symptom or risk
A set that depends on one sample is presented as the drivers of the outcome, and next quarter's refit disagrees.
Exceptions
Very small samples, where half-samples are too thin for any stable selection.
Trade-offs
Costs many refits; the threshold is a trade between false inclusions and missed features.
Pitfalls
Running the subsampling on all rows before the train-test split, which leaks the test rows into selection.
Lessons and concepts
Claims and their sources
Stability selection combines subsampling with high-dimensional selection algorithms and provides finite-sample control of some false discovery error rates.