With 128k context windows, I see people just stuffing entire books into the prompt. It seems like most RAG systems are badly designed because they rely on the model to "find the needle." Does anyone actually get better results with RAG than just using a massive context window?
3 answers
The "Long Context vs RAG" debate is a classic example of why most RAG systems are badly designed—they are often used when a simple long context would suffice, or vice versa. However, RAG is still superior for three reasons: cost, latency, and "Lost in the Middle" issues. Even the best models lose accuracy when the relevant info is buried in the middle of a 100k token prompt. A well-architected RAG system only sends the most pertinent 2k tokens, which is cheaper and faster. Designing for RAG means you are optimizing for the model's "attention" by removing the noise before the prompt is even built.
Heather, at what token count does the cost of managing a vector database and RAG infrastructure become higher than just paying the extra API fees for a long context window?
RAG is for scaling. You can't put 10 million documents in a context window, so RAG is the only logical design for big data.
Spot on, Edward. Long context is a temporary fix, but RAG is the long-term infrastructure for any serious AI application.
Scott, it’s not just the money; it’s the speed. If your agent takes 30 seconds to read a "book" every time a user asks a question, the UX is dead. We found that for any dataset over 50,000 tokens that needs frequent querying, the investment in a RAG pipeline pays off in performance within the first month.