We are moving our production data to Milvus and need a solid backup strategy. What are the best tools for taking snapshots of a distributed cluster? If a data node fails, how quickly can we recover without losing our vector indexes? Is there a way to do incremental backups?
3 answers
For a distributed setup, you should use the Milvus Backup tool provided by the community. It allows you to create snapshots of your collections and store them in S3-compatible object storage. Since Milvus separates storage and computing, your actual data is stored in MinIO or S3 anyway. If a data node fails, the coordinator will automatically reassign the data segments to other healthy nodes, so you don't actually lose data just from a node failure. The backup tool is more for accidental deletions or total cluster failures. For incremental backups, you can monitor the binlogs, but it is much simpler to just snapshot the underlying object storage buckets.
Have you tested the restoration speed with a large dataset? Sometimes re-indexing the whole snapshot can take hours if the HNSW parameters are complex.
The Backup tool is definitely the most reliable way. It captures the metadata and the actual vector data together so the state remains consistent.
Natalie is right. Just make sure your MinIO/S3 permissions are correctly configured or the backup script will fail silently halfway through.
That is my biggest fear. We have about 50 million vectors. If we have to rebuild the index from scratch during a recovery, our search service will be down for way too long. We are looking into maintaining a "warm" standby cluster in a different region and using the backup tool to keep it synced every hour to minimize potential downtime.