I'm using K-Means for market segmentation, but the clusters look very "forced" and don't make much business sense. How do I determine the optimal number of clusters (K) beyond the Elbow Method, and what are the signs that my features need better normalization?
3 answers
The Elbow Method is often ambiguous. I highly recommend using the "Silhouette Score"—it measures how similar an object is to its own cluster compared to others. If your segments look "forced," it’s often because of outliers or skewed distributions. K-Means is very sensitive to these. In my work with a travel agency, we found that normalizing with "Min-Max Scaling" wasn't enough; we had to use "Log Transformation" for income data because the high-earners were pulling the cluster centers too far. Always visualize your clusters using PCA (Principal Component Analysis) to see if they actually separate in space.
Make sure your features are on the same scale! If one variable is "Income" (thousands) and another is "Age" (tens), the algorithm will almost entirely ignore Age.
Have you tried using a Density-Based method like DBSCAN instead? It’s much better at finding irregular shapes and handling noise than K-Means is.
James, I tried DBSCAN, but the "epsilon" parameter is a nightmare to tune with high-dimensional data. However, it did help me identify a group of "outlier" customers that K-Means was trying to hide inside other clusters. Now, I use a two-step process: I use DBSCAN to clean out the noise first, and then I run K-Means on the "dense" core of the data. This gives much cleaner, more actionable segments for our email marketing campaigns.
Exactly, Sarah. Scaling is the #1 reason for "weird" clusters. Standardizing the variance to 1 is a mandatory step before you even touch the clustering library.