Our search engine needs to index thousands of 50-page PDF reports. When we index them as single documents, the retrieval is poor because the chunks are too large for the LLM context. What are the best practices in Haystack for "Chunking" and "Preprocessing" long PDFs to ensure the search engine finds the exact paragraph needed?
3 answers
For long PDFs, the PreProcessor component in Haystack is your best friend. You should use the split_by="word" or split_by="sentence" options with a split_overlap of about 10-15%. This ensures that no context is lost at the boundaries of the chunks. In a project I did in late 2023, we found that a split_length of 250 words worked best for our technical manuals. This size is small enough to provide very specific context to the LLM but large enough to retain a coherent meaning. Also, make sure to enable clean_whitespace to remove all those messy PDF artifacts.
When you split a 50-page PDF into smaller chunks, how do you handle the loss of global context? For example, if a chunk on page 40 refers to "The Project" mentioned on page 1, the retriever might find the chunk but the LLM won't know what project it's about.
Using the PyPDFToDocument converter with a well-configured PreProcessor handles 90% of PDF issues. It’s all about finding that "Goldilocks" chunk size for your specific data.
Definitely. I’ve found that experimenting with different split lengths can lead to a 20% increase in retrieval accuracy. It’s worth the extra testing time.
Gregory, that's where "Metadata Injection" comes in. When you run the PreProcessor, you can tell Haystack to attach the document's title or summary to every single chunk. This way, even if a chunk is deep in the report, the LLM still sees the context that it belongs to "Project X." It’s a very common SEO and RAG strategy to ensure the "local" chunk carries "global" meaning. Haystack makes this automated during the indexing pipeline phase.