We are migrating from a legacy keyword search to a vector-based search using Chroma DB. We have over 500,000 records. What is the most efficient strategy to ingest this data without crashing the server or causing downtime for our current users? Should we use a staged approach?
3 answers
For a dataset of 500k records, a staged ingestion is the only safe way to go. I recommend using a worker-queue pattern. Push your records into a queue like RabbitMQ or Redis, and have multiple worker processes handle the embedding and insertion into Chroma DB in batches of 100 to 500. This prevents memory spikes on your main server. You should build the new index on a separate "shadow" instance. Once the shadow instance is fully synced and tested, you can perform a simple DNS or load-balancer switch to point your production traffic to the new Chroma DB setup seamlessly.
If we use a shadow instance, how do we handle the delta of new records created during the migration period?
I've found that pre-calculating embeddings locally before sending them to the server saves a ton of CPU resources on the database machine itself.
Exactly, Rebecca. Offloading the heavy lifting of embedding generation makes the actual database ingestion much faster and more stable.
Patrick, you should implement a "dual-write" strategy. Once you start the migration, your application should write all new incoming data to both the legacy system and the new Chroma DB shadow instance. This way, by the time your bulk migration of the old 500k records is finished, the shadow instance is already up-to-date with the most recent entries, allowing for a clean and perfectly synced cutover.