Everyone focuses on the LLM, but is our data pre-processing the real reason why RAG systems feel badly designed? If the chunking is wrong, the agent gets fragments of sentences that make no sense. How do we fix this?
3 answers
You are spot on. Most developers use "fixed-size" chunking, which is the equivalent of cutting a book into pieces every 500 characters, regardless of where the paragraph ends. This destroys the semantic meaning. For a high-performing agent, you need "semantic chunking" or "recursive character splitting." This method ensures that code blocks, lists, or related paragraphs stay together. If the agent receives a complete thought rather than a sentence fragment, its ability to reason and provide a correct answer increases exponentially. This is a Data Science fundamental that many AI engineers are currently ignoring.
Scott here. Is there a specific overlap percentage you recommend between chunks to ensure the context isn't lost at the boundaries?
I think people should look into "Small-to-Big" retrieval, where you search small chunks but send the larger surrounding paragraph to the LLM.
Rebecca, that is a brilliant architectural pattern. Jeffrey, this specifically solves the "fragmented sentence" problem by providing a broader context for the answer.
Scott, a 15-20% overlap is usually the sweet spot. It provides enough redundant context for the embedding model to "see" the connection between adjacent chunks without wasting too much space in the context window. It acts like a safety net for the retrieval engine.