We are trying to automate our accounts payable using Multi-Modal LLMs like GPT-4o. The challenge is that about 30% of our invoices are hand-written or have very messy stamps over the text. Traditional OCR often fails here. How should I structure my "vision prompts" to ensure the model doesn't hallucinate the numbers? Should I tell the model to "transcribe everything first" or can I just ask it to "output a JSON of the total amount"? I need a prompt strategy that is robust against low-quality images.
3 answers
For messy documents, you must use a "Chain-of-Visual-Thought" approach. Don't ask for the JSON immediately. Instead, prompt the model to: 1. "Describe the layout of the document." 2. "Identify all hand-written sections." 3. "Transcribe the line items one by one." 4. "Finally, calculate the sum and output as JSON." By forcing the model to "see" and "verbalize" the components first, you significantly reduce the chance of it jumping to a wrong total. Also, explicitly tell it: "If a number is illegible, write [UNCERTAIN] rather than guessing."
Have you tried comparing the results of different models on the same image? I’ve found that some models are better at reading cursively written text while others are better at structured tables.
Always include a "negative prompt" equivalent. Tell the model to ignore background noise, coffee stains, or "PAID" stamps that might obscure the actual numbers in the invoice.
Great point, Michael. Those "PAID" stamps are the number one cause of OCR errors in our experience. Specifically telling the AI to "look behind" the stamp works surprisingly well.
Charles, that's a pro tip. We actually use a "Voter" prompt strategy. We send the image to both GPT-4o and Claude 3.5 Sonnet. A third "Judge" prompt then compares the two outputs. If the totals match, it goes to the database. If they don't, it gets flagged for a human. It’s a bit more expensive, but for financial data, that 99.9% accuracy requirement makes the extra cost worth it. It’s much cheaper than a human manually typing in 1,000 invoices a day.