We are seeing latency spikes when our user base grows. Does anyone have tips for tuning Milvus to handle thousands of concurrent queries? Specifically, we are debating between using the IVF_FLAT or HNSW index types. Which one offers the best balance of speed and recall when scaled across multiple nodes in a Kubernetes cluster?
3 answers
In my experience, HNSW is generally superior for high-concurrency needs because it provides much faster query speeds despite a slightly higher memory footprint. When deploying on Kubernetes, ensure you are utilizing the Milvus Operator to manage your components. You should specifically look into horizontal scaling of the Query Nodes, as these are responsible for the actual search execution. Also, consider implementing a caching layer like Redis in front of your database for the most frequent queries to reduce the load. Lastly, make sure your segment sizes are optimized; having too many small segments can severely degrade search performance during high traffic periods.
What kind of hardware are you running your Query Nodes on? Are you utilizing GPUs for index acceleration or sticking strictly to CPU-based processing?
I found that tweaking the nq and topk parameters helped stabilize our latency. Sometimes the bottleneck is actually the network throughput between nodes.
Great point, Kimberly. I actually saw a 15% improvement in our environment just by upgrading the node-to-node networking bandwidth in our cluster configuration.
We are currently on CPU-only nodes using high-memory AWS instances. I’ve heard that GPU acceleration can significantly boost throughput for certain index types, but we are worried about the cost-to-performance ratio. Have you seen a significant enough jump in queries per second to justify the switch to expensive NVIDIA instances for a standard RAG application?