We are exploring Chroma DB as a potential backend for a real-time recommendation engine we're developing. My main concern is how it handles frequent upserts. If we are constantly updating user preference vectors, will the indexing speed of the vector database become a bottleneck? We need something that can handle relatively low latency for both writes and similarity searches. Is this tool robust enough for a system with a few thousand active users, or should we be looking at enterprise-grade alternatives?
3 answers
For a few thousand users, Chroma should handle the load quite well, especially if you utilize the upsert method correctly. It’s designed to be lightweight and fast. The HNSW algorithm it uses is very efficient for incremental updates without needing a full re-index of the collection. However, you should monitor your memory usage. Since Chroma is optimized for speed, it tends to keep a lot of the index in RAM. As long as your server has sufficient memory to hold the embeddings and the index structures, the latency for similarity searches should remain in the millisecond range
Have you considered how you will handle high-dimensional vectors? Sometimes the performance hit comes from the embedding size rather than the database itself.
It’s definitely robust enough for a prototype or a medium-scale app. The open-source nature makes it easy to debug and tweak for specific ML needs.
I agree with George. For a system with a few thousand users, the simplicity of setting up Chroma DB often outweighs the overhead of more complex systems.
I've noticed that too. If you are using 1536-dimensional vectors from OpenAI, the search is naturally slower than with 384-dimensional ones. Chroma DB performs well, but reducing dimensionality can help.