I am working on a data pipeline that scrapes millions of articles daily. We are seeing a lot of near-duplicate content. Can Qdrant help identify these duplicates using embeddings? I'm curious if I can set a similarity threshold to flag articles that are 95% similar and how the system performs when thousands of new items are added every hour.
3 answers
Using Qdrant for deduplication is a very common and effective use case. By converting your articles into dense vectors using a model like Sentence-BERT, you can upsert them into Qdrant. You can then use the "Search" API with a 'score_threshold' to find any existing articles that are too similar to the new one. Since Qdrant is optimized for high-write throughput, it can easily handle thousands of upserts per hour. We use a similar setup for a news aggregator and it has significantly cleaned up our feed. The ability to update the payload without re-indexing the vector is also a huge plus for tracking sources.
How do you handle the "cold start" problem when the index is being rebuilt? Does Qdrant allow for concurrent searching and indexing without the searches becoming sluggish?
You should definitely use the "Recommendation API" in Qdrant too. It can find items similar to "positive" examples and far from "negative" ones, which helps refine the deduping logic.
I never thought of using the Recommendation API for deduplication, Heather! That’s a clever way to handle edge cases where simple similarity isn't enough.
Andrew, Qdrant uses a write-ahead log (WAL) and segment-based indexing. New data is first written to a "memmap" segment that is immediately searchable. Later, it gets merged into the larger HNSW segments in the background. This architecture allows you to keep searching at high speeds even while you are hammering the database with new data. You might see a slight CPU spike during the segment merging, but the search latency is generally unaffected.