Classical ML, validation, metrics, interpretation, debugging · insight-0192
Do not assume the top principal components carry the predictive signal
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- Data scientist, ML engineer
- Industries
- manufacturing, energy
Summary: situation, action, reason, boundary
Situation
You reduce features with PCA before a supervised model and keep components explaining 95% of variance.
Action
Compare PCA-reduced and raw-feature models on held-out data, tune the number of components by validation score rather than explained variance, and consider regularization or supervised reduction instead.
Reason
PCA picks directions of maximum variance in X without looking at y, so variance and relevance need not coincide.
Boundary
Principal components regression, PCA before classifiers, and compression for speed.
Symptom or risk
A low-variance direction that predicts the target is discarded; the model underperforms a model on the raw features.
Exceptions
When noise dominates many correlated features and the signal lies in shared variation, PCA can help.
Trade-offs
Fewer components speed models and reduce collinearity, at the cost of interpretability and possibly signal.
Pitfalls
Fitting PCA on all rows before cross-validation, which leaks the test distribution.
Lessons and concepts
Claims and their sources
Principal components are identified without the response, so there is no guarantee that the directions that best explain the predictors also best predict the response (ISL section 6.3.1).