We are redesigning our e-commerce search to be "AI-First." I want users to be able to upload a photo of a product and ask, "Do you have this in blue and in my size?" What is the best way to architecture this using multimodal models like Gemini Pro Vision or GPT-4o? How do we index our visual catalog so the AI can compare live images against our inventory in real-time?
3 answers
To build a true multimodal search, you need to move beyond traditional keyword tagging. You should use a "Unified Embedding Space" where both your product images and their text descriptions are converted into vectors using the same model (like CLIP or a specialized multimodal encoder). When a user uploads a photo, you convert that image into a vector and perform a "Nearest Neighbor" search in your database. Then, you pass the top results plus the user's text prompt ("in blue and my size") to a multimodal LLM to filter the final selection. This allows the system to understand the visual style while respecting the specific text-based constraints.
Do you find that the latency for processing an image plus a text query is too high for a snappy "live" search experience on mobile devices?
Don't forget about "Visual Grounding." You want the AI to be able to point to specific features in the user's photo to explain why it recommended a certain product.
Exactly! Visual grounding builds so much trust. If the AI can say "I found this because the lapel style matches your photo," the customer is much more likely to buy.
It was a problem initially, taking nearly 5 seconds. We optimized by doing the "visual similarity" check first on a smaller, local index to narrow it down to 20 items, and only then sending those 20 items and the user prompt to the heavy cloud-based multimodal LLM for the final answer. Now the whole process feels almost instantaneous for the end-user.