With the rise of RAG (Retrieval-Augmented Generation), it seems like vector databases are becoming a core part of the Big Data stack. Is anyone successfully integrating Pinecone or Milvus with their existing Hadoop/Spark clusters? I'm trying to figure out if we should be treating "Embeddings" as just another column in our Parquet files or if they need a specialized system.
2 answers
You definitely shouldn't just store them in Parquet if you want to perform similarity searches at scale. While you can store a 1536-dimensional vector as an array in Parquet, searching through a billion rows using a standard Spark scan would take minutes. Specialized Vector Databases like Milvus or Weaviate use HNSW indexing to do that search in milliseconds. In our stack, we use Spark to generate the embeddings in a batch job (using a model like Titan or Ada), and then we "sink" those vectors into Milvus. This way, our Big Data pipeline handles the "heavy lifting" of data preparation, while the Vector DB handles the real-time AI retrieval for our chatbot.
Doesn't that create another "Data Silo"? Now you have your business data in a Warehouse and your "Context" in a Vector DB. How do you join a customer's purchase history with their support ticket embeddings without doing a slow cross-system call?
Mark, that’s exactly why Snowflake and Databricks are building "Native Vector Support." Databricks now has "Vector Search" built directly into the Unity Catalog. You can define a table and tell Databricks to automatically maintain an index for a specific column. This keeps everything in one place. If you're already on a modern Lakehouse platform, I'd check if they have a native vector index feature before jumping into a standalone system like Pinecone, as it simplifies governance significantly.