We are scaling our recommendation engine and the storage costs for our vector database are getting out of hand. We currently store 100 million vectors with 768 dimensions each in float32 format. Are there any reliable compression or quantization techniques that won't significantly degrade our search accuracy? I've heard of Product Quantization (PQ), but I'm worried about the performance hit.
3 answers
Scalar Quantization is the easiest fix. It cuts memory use by 75% and you probably won't even notice the change in your recommendation quality.
You should definitely look into Scalar Quantization (SQ8). By converting your float32 vectors (4 bytes per dimension) to int8 (1 byte), you can reduce your memory footprint by 4x with very little impact on recall—usually less than 1-2%. If you need even more compression, Product Quantization (PQ) is the way to go, though it is more complex to tune. PQ breaks the vector into sub-vectors and clusters them, allowing you to store a massive dataset in a fraction of the space. Most managed services like Zilliz or Qdrant have built-in toggles for these, making the transition relatively painless.
Have you considered using a smaller embedding model? Sometimes a 384-dimension model gives nearly the same results as 768 but halves your storage immediately.
Kevin, that's a bold move but often effective. However, if they already have millions of vectors, re-embedding everything is a huge task. A better immediate step for Mary might be "Disk-based indexing." Some vector databases allow you to store the bulk of the index on SSD instead of expensive RAM. With algorithms like DiskANN, you can maintain single-digit millisecond latency while cutting costs by 70-80%. It's the standard approach for "billion-scale" search where keeping everything in memory is simply not financially viable for most companies.
I'll look into SQ8 first, Nancy. A 4x reduction in memory sounds like exactly what our CFO wants to hear right now!