Feature engineering and preprocessing · insight-0102
Scale features before k-NN, SVMs and k-means or the largest unit decides the distance
- Evidence type
- documented fact
- Difficulty
- easy
- Roles
- Data scientist, ML engineer
- Industries
- aviation
Summary: situation, action, reason, boundary
Situation
A distance- or kernel-based model uses features in mixed units, such as average fare in currency, load factor in 0-1 and days to departure in days.
Action
Put a scaler fitted on training folds inside the pipeline, compare StandardScaler and RobustScaler, and inspect which features drive nearest-neighbour lists after scaling.
Reason
Euclidean distance and RBF kernels sum squared differences, so a feature with variance a thousand times larger dominates the objective and starves the others.
Boundary
k-NN, SVMs with RBF or polynomial kernels, k-means, DBSCAN, PCA and gradient-trained neural networks.
Symptom or risk
Neighbours and clusters are determined almost entirely by the feature with the largest numeric range; the other features look useless.
Exceptions
If all features share a meaningful common unit (for example, pixel intensities), rescaling can remove information you want the distance to keep.
Trade-offs
Equal scaling is itself a weighting choice; domain weights may beat it but must be justified and validated.
Pitfalls
Fitting the scaler on the full dataset before splitting, which leaks test statistics into training.
Example
Synthetic airline data: in a route table of average fare, load factor, capacity and booking-curve shares, average fare holds about 95% of the total variance when unscaled, so k-means clusters mostly on fare.
Lessons and concepts
Claims and their sources
RBF-kernel SVMs and l1/l2-regularized linear models may assume centered, similarly scaled features; a feature with much larger variance can dominate the objective.