I have a dataset with over 200 features. I was taught that Principal Component Analysis (PCA) is the go-to method for dimensionality reduction, but I'm worried about losing the physical meaning of my variables. When is it appropriate to use PCA versus just performing feature selection based on correlation matrices?
3 answers
PCA is excellent when you have high multicollinearity—meaning your features are highly correlated with each other. It "squashes" that information into orthogonal components. However, the downside is exactly what you mentioned: loss of interpretability. You can't tell a stakeholder that "Component 1" drove the prediction. If interpretability is your priority, stick to feature selection using Lasso (L1) regression or Feature Importance from a Tree-based model. Use PCA mainly when you are struggling with the "Curse of Dimensionality" or when you need to speed up the training of distance-based models like KNN or SVM.
Have you looked at the "Explained Variance Ratio" to see how many components you actually need to keep 95% of the information?
If you need to stay interpretable, try UMAP or t-SNE for visualization but keep the original features for the actual predictive modeling.
Julia is right. Visualization and modeling are two different goals. Use the complex stuff for sight and the simple stuff for the logic.
I did check the scree plot, Philip. It looks like I can get away with just 15 components to keep 90% variance. My concern is still the business side of things; if the model rejects a loan, I need to explain why. Can I back-calculate the influence of the original features from the principal components easily enough to satisfy a regulatory audit?