Feature engineering and preprocessing · insight-1131
Report selection stability across folds as an overlap statistic next to the accuracy
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist
- Industries
- manufacturing, healthcare
Summary: situation, action, reason, boundary
Situation
Two selection methods reach the same cross-validated score, and you must choose one to document as the feature recipe.
Action
Record the selected subset on each training fold, compute the average pairwise overlap (Jaccard or a stability index), and prefer the method with higher stability among equally accurate ones.
Reason
Accuracy ignores which features were chosen; stability measures how sensitive the selection is to training-data changes.
Boundary
Wide data with hundreds of candidate features, such as sensor or claims data.
Symptom or risk
The method whose chosen subset changes completely from fold to fold is picked on a coin flip, and the story told about the features does not replicate.
Exceptions
Pure prediction where only the score matters and the feature set is never reported or reused.
Trade-offs
Highly correlated features reduce stability even when they are interchangeable, so group them before measuring.
Pitfalls
Comparing overlap between subsets of different sizes without a correction for chance overlap.
Lessons and concepts
Claims and their sources
Kalousis, Prados and Hilario study the stability of feature selection algorithms, meaning sensitivity to training-data variation, compare several stability measures and use stability profiles to choose an algorithm.