I've heard about people using Kafka to feed vector databases like Milvus or Pinecone. But can Kafka itself serve as a vector store for simple RAG (Retrieval-Augmented Generation) applications? How would one implement a similarity search across a Kafka topic containing millions of embeddings?
3 answers
While Kafka isn't a native vector database, you can implement a "lite" version using Kafka Streams and a custom state store. You would store your embeddings in a ReadOnlyKeyValueStore. To perform a similarity search (like Cosine Similarity), you would iterate through the state store records. However, because Kafka state stores are key-value based, this is an $O(n)$ operation, which is slow for millions of vectors. For production RAG, it's better to use Kafka as the "source of truth" and use Kafka Connect to sync the embeddings into a dedicated vector DB that supports HNSW indexing for $O(\log n)$ search speeds.
Could we use a custom interactive query to partition the vectors so each Kafka pod only searches its own subset, effectively parallelizing the similarity search?
I think the best use of Kafka here is the "Streaming Ingestion" part. Use Kafka to handle the high-velocity ingestion and then let a specialized DB handle the heavy math.
I agree with Diana. Kafka is the perfect "buffer" and "sequencer" for your data before it hits the vector search engine.
Patrick, yes! That is actually how distributed vector databases work under the hood. By using Kafka’s partitioning, each instance of your application only holds a fraction of the data in its local RocksDB. You could send a broadcast query to all instances, aggregate the top "K" results from each, and then pick the global top results. It’s a fun engineering challenge, but unless you have very specific requirements, a tool like Weaviate or Milvus already does this much more efficiently than a custom-built Kafka implementation.