We are trying to automate invoice processing where the layouts are completely unstructured and change daily. Standard OCR isn't cutting it. Has anyone successfully integrated an LLM like GPT-4 into their RPA pipeline for data extraction, and how are you handling the API costs and data privacy concerns?
3 answers
We recently launched a pilot using "Intelligent Document Processing" (IDP) combined with an Azure OpenAI instance. To solve the privacy issue, we used an enterprise-tier API where our data is not used for model training. The trick to managing costs is not sending the whole document. We use a standard OCR to extract text first, and then send only the "ambiguous" segments to the LLM for classification. This reduced our token usage by 70%. It has brought our accuracy for handwritten notes from 40% to nearly 95%. It’s definitely the future of "Hyperautomation," but you must have a human-in-the-loop for final validation.
Are you finding that the LLM's response time creates a bottleneck in your real-time processing bots, or do you run these as batch jobs?
We use a local Llama-2 model on our own servers to avoid any data leaving our network. The accuracy is slightly lower than GPT-4, but the privacy is 100%.
I agree with Deborah. For highly sensitive data, local models are the way to go. Even if they are 5% less accurate, the lack of compliance risk is a huge win for the legal department.
Gregory, we definitely run them as batch jobs. The latency of an LLM call can be 5-10 seconds, which is a lifetime for a synchronous bot. By offloading the extraction to a background queue, our main bots can keep moving. We then have a separate "Review Bot" that aggregates the LLM results for the human auditors. This asynchronous pattern is essential if you want to scale AI-integrated RPA without killing your server performance.