I'm exploring the use of Chroma DB for an anomaly detection system in our cyber security department. We want to convert system logs and network traffic patterns into embeddings to find "similar" historical threats. Given the sheer volume of logs we generate daily, I'm worried about how the database handles multi-tenancy and if it can support billions of vectors without significant latency. Has anyone used a vector database for this kind of high-frequency log analysis?
3 answers
Using a vector database for threat hunting is an excellent strategy. For Chroma DB, the challenge with "billions" of vectors will be the horizontal scaling. While the latest versions have made strides in multi-tenant support, a single-node setup will likely struggle with that volume. You would need to shard your collections across multiple instances or look into their distributed deployment options. That said, for localized threat intelligence—like comparing the last 24 hours of logs against known attack signatures—it’s incredibly fast and much more effective than keyword-based searching.
What kind of embedding model are you using for the log data? Standard NLP models might not capture the nuances of network protocols very well.
For massive datasets, ensure you're using the HNSW index with optimized parameters like M and ef_construction to balance speed and recall.
Solid advice, Helen. Tuning those HNSW parameters within Chroma DB is essential for maintaining performance as your log volume grows.
We are actually training a custom model on our historical log data to generate specialized embeddings. Once we have those, we plan to pipe them into Chroma DB. The main goal is to identify patterns that don't match any known "good" signatures. It's more about the similarity to past attacks than a direct match, which is why we're moving away from traditional SIEM rules for this specific use case.