We’ve been running Chroma DB in a Docker container using persist_directory for our internal knowledge base. It worked great for the first 50k documents, but now our query latency is spiking to 500ms+ as we approach 1M vectors. Is it worth moving to the new Chroma Cloud managed service, or should we just throw more RAM at our self-hosted instance?
3 answers
If you're seeing 500ms latency, you're likely hitting the limits of the single-node SQLite backend used in the older open-source versions. Chroma Cloud uses a distributed architecture that separates storage from query execution. In my migration last month, our p95 latency dropped from 450ms to about 45ms. If you stay self-hosted, make sure you've upgraded to the latest 2025/2026 releases which feature the Rust-core rewrite—it handles multi-threading much better than the old Python-heavy implementation.
Does the Cloud version support the same metadata filtering syntax, or do we need to rewrite our where clauses?
I suggest checking your hnsw:space parameter. Switching from 'cosine' to 'l2' can sometimes give a slight speed boost if your embedding model supports it.
Good tip, Charles. Though for 1M+ vectors, the managed distribution of Chroma Cloud is usually a safer bet for maintaining that sub-100ms SLA without a dedicated DevOps person.
It’s a 1:1 API match, Kevin. That’s the best part—you just change your HttpClient initialization to point to the Cloud tenant and add your API key. No code refactoring is required.