I'm building a semantic search engine using Sentence-BERT. I've generated my embeddings, but I'm getting different results depending on the distance metric I use. For high-dimensional text vectors, which one is considered the "gold standard" for measuring document similarity?
3 answers
For NLP and high-dimensional embeddings, Cosine Similarity is almost always the preferred choice. The reason is that Cosine Similarity measures the angle between two vectors, rather than the magnitude. In text, one document might be much longer than another but discuss the same topic. Euclidean distance would penalize the longer document because its vector is "further" from the origin due to word frequency. Cosine Similarity ignores this length factor and focuses purely on the direction of the semantic content. If you are using normalized vectors (where magnitude is 1), both metrics actually become mathematically equivalent, but Cosine remains the industry standard for search.
Does this change if I'm using a specific vector database like FAISS or Milvus? Some of them seem to optimize for Inner Product (IP) over Cosine.
If you are doing clustering (like K-Means) on your text data, Euclidean distance is sometimes preferred by the algorithm, but for search/retrieval, stick to Cosine.
Exactly, Karen. It really depends on the "Downstream Task." Search is about orientation (Cosine), while clustering is often about density and distance (Euclidean).
Scott, you're right about the optimization. Many vector databases prefer Inner Product because it’s computationally faster to calculate on hardware. However, if you normalize your vectors to a length of 1 during the preprocessing stage, the Inner Product is the Cosine Similarity. So, the trick is to L2-normalize your Sentence-BERT outputs before indexing them. This way, you get the speed of the Inner Product calculation in FAISS while maintaining the semantic accuracy of the Cosine metric. It’s a common performance hack in production-scale search engines.