I am noticing that newly inserted data isn't immediately searchable. How do I ensure better consistency in Milvus without sacrificing ingestion speed? We are running a daily ETL pipeline that pushes millions of vectors, and our downstream applications need to access this data as soon as the "upsert" call returns a success.
3 answers
Milvus follows a "tuned consistency" model. By default, it might be set to 'Bounded' or 'Session' consistency. If you need immediate searchability, you should set the consistency level to 'Strong'. However, be warned that this will significantly increase the latency of your write operations because the system must ensure all nodes are synced before confirming the write. For batch ETL, a better approach might be to use the 'bulk_insert' feature which is much faster than individual upserts. Once the bulk load is finished, you can call 'flush' and 'load' to ensure the segments are indexed and ready for the query nodes to pick up.
Are you using the standalone version or the distributed cluster mode? The way the Pulsar or Kafka log works can impact how quickly data reaches the queryable state.
You might want to check your 'flush' settings. If your segments stay in 'growing' state too long, they aren't indexed, which makes searches much slower.
Exactly, Ryan. Forcing a flush after a large batch is a manual but effective way to trigger the indexing process immediately.
We are using the distributed mode with Pulsar. I noticed that during peak ingestion, the log sequence seems to lag. Is there a specific configuration in the data nodes that I should look at to speed up the transition from the log to the sealed segments? We really need that data live within a few seconds of ingestion for our news monitoring tool.