Is the reliance on vector embeddings the reason why most RAG systems are considered poorly built?
3 answers
The "badly designed" label comes from ignoring keyword-based search. Vector embeddings are great for nuance but terrible for specific technical terms or part numbers. A well-designed system must utilize hybrid search—combining BM25 or traditional keyword indexing with dense vector retrieval. This ensures that when a user looks for a specific term, the system doesn't just find "something similar" but finds the exact match. Furthermore, the lack of an evaluation framework like Ragas makes it impossible for developers to know if their system is actually improving or just guessing.
Have you tried implementing a reranker like Cohere to see if that solves the accuracy issues you are seeing in your current setup?
Hybrid search is definitely the answer. Combining traditional SQL-like queries with vector search fills the gaps that most people complain about.
I agree with Laura. Most "bad" designs are just incomplete designs. Adding a lexical search layer usually fixes 80 percent of the retrieval errors instantly.
Joshua, rerankers are a game changer. They take the top 20 results and perform a more intensive computation to pick the best 3. It adds a bit of latency, but the quality jump is usually worth the extra few hundred milliseconds in a production environment.