I’ve been testing several retrieval-augmented generation pipelines, and I’m beginning to wonder: are most RAG systems badly designed from the start? Many seem to struggle with semantic drift and poor indexing, leading to irrelevant or hallucinated answers. Is there a fundamental flaw in how we are architecting these systems for enterprise data?
3 answers
The assertion that most RAG systems are badly designed often stems from a "naive RAG" approach where developers simply dump text into a vector store. The real problem lies in the lack of sophisticated pre-processing and post-retrieval steps. To fix this, you must implement recursive character splitting, metadata filtering, and most importantly, a re-ranking step using a Cross-Encoder. Without these, the system retrieves top-k chunks based on keyword similarity rather than semantic relevance. A well-designed system treats retrieval as a multi-stage pipeline, ensuring the context provided to the LLM is precise, concise, and contextually grounded.
Kimberly, do you believe the reliance on simple cosine similarity is the main reason why most RAG systems are badly designed? Would switching to a hybrid search model using BM25 and vector embeddings solve the majority of these grounding issues in production?
The design fails when people ignore data cleaning. Bad data in means bad retrieval out, regardless of how advanced your embedding model is.
Totally agree with Kevin. We spent weeks tuning our vector database only to realize our source PDFs were formatted so poorly that the parser was creating gibberish.
Brian, you hit the nail on the head. Hybrid search is a massive leap forward. However, the design failure often goes deeper into the "small-to-big" retrieval strategy. If you don't retrieve the parent context surrounding a child chunk, the LLM misses the forest for the trees. Addressing this hierarchy is what separates a toy project from an enterprise-grade RAG solution.