We are trying to build a system that searches both images and text. Does Chroma DB support multi-modal embeddings natively, or do we need to store different types in separate collections? I want to know the most efficient way to query a text prompt and get back the most relevant images.
3 answers
Chroma DB is very flexible with multi-modal data. The most efficient approach is to use a unified embedding model like CLIP, which can map both images and text into the same vector space. You don't need separate collections; you can store the image embeddings and text embeddings together. When a user submits a text query, you embed it using the same CLIP model and run a similarity search in Chroma. The results will return the closest vectors, which could represent either images or text, effectively bridging the gap between different data types in a single query.
If we store everything in one collection, how do we handle the metadata to filter specifically for images when we only want visual results?
I've implemented this using the OpenCLIP integration. It works seamlessly with Chroma's collection.query method for cross-modal retrieval.
Justin's suggestion is the way to go. OpenCLIP offers various model sizes that let you balance between search accuracy and processing speed.
Brandon, you can easily handle this by adding a "media_type" key to your metadata dictionary during the upsert process. When querying, use the where clause in Chroma to filter for {"media_type": "image"}. This ensures that even though your search is happening across the entire vector space, your results will only contain the data types you've specified for that particular request.