Most RAG systems seem to be "set and forget." Is this why they are perceived as badly designed? Without a way for users to "thumbs up" or "thumbs down" the retrieved context, how can the system ever improve its ranking logic?
3 answers
You hit the nail on the head. A static RAG system is a dying system. Modern software development practices require observability. If you aren't logging what was asked and what was retrieved, you can't perform "fine-tuning" on your embedding model or adjust your weights. A "well-designed" system uses Reinforcement Learning from User Feedback (RLHF) or at least basic telemetry to identify which documents are consistently failing to answer questions. Without this, the system is just a black box that gradually loses its utility as the data grows.
How would you suggest we store this feedback data so it can be used to automatically update the vector index weights?
I think simple analytics can go a long way. Just seeing what people search for most helps you refine the knowledge base.
Martha is right. Often, the design is fine, but the "content gap"—where users ask for things not in the data—is the real culprit.
Patrick, you can use a relational database to map query IDs to document IDs and the user score. Then, you can periodically run a script to boost the "relevance score" of highly-rated documents in your metadata, ensuring they appear higher in future search results.