I am using KNN for a recommendation system. I noticed that the results change drastically when I switch from Euclidean distance to Cosine similarity. Why does this happen, and how do I determine which metric is mathematically appropriate for high-dimensional text data?
3 answers
The choice of metric depends on whether "magnitude" matters. Euclidean distance measures the straight-line distance between two points; it’s sensitive to the scale of your features. If one user buys 100 items and another buys 1, Euclidean will say they are very different even if they bought the same types of items. Cosine similarity measures the "angle" between vectors, ignoring magnitude. For text or recommendations, Cosine is usually better because it focuses on the pattern of the data rather than the volume. In 2024, for high-dimensional spaces, Cosine is the standard because Euclidean distance starts to lose its meaning as dimensions increase.
Do you always perform Min-Max scaling or Z-score normalization before applying Euclidean distance, or does it depend on the feature distribution?
For sparse data like word counts, Manhattan distance (L1) can sometimes outperform Euclidean (L2) because it is less sensitive to outliers.
Good point, Samantha. L1 is definitely more robust when you have noisy data or many zeros in your matrix.
Dustin, scaling is mandatory for Euclidean distance. If you don't scale, a feature with a range of 0-1000 will completely dominate a feature with a range of 0-1. For Cosine similarity, it’s less critical since it’s scale-invariant to some degree, but I still recommend it as a best practice. I learned the hard way that skipping the scaling step makes your KNN model essentially act as a 1-feature model.