With the release of models like GPT-4o and Gemini 1.5 Pro, we now have native "Multimodality." I’m curious how this changes the pipeline for people working in Data Science. Before, if I wanted to analyze a chart or a medical scan, I’d need a specialized Computer Vision model. Now, I can just "show" the image to the LLM. Is this reliable enough for professional data extraction from complex tables and graphs, or should we still rely on OCR-specific tools like Amazon Textract or specialized CNNs?
3 answers
Multimodal LLMs are a game-changer for "semantic" understanding of images. They are much better at explaining why a trend is happening in a graph compared to a traditional OCR tool. However, for "spatial" precision—like extracting the exact pixel coordinates of a small data point—they can still struggle. If your task is extracting data from 10,000 structured invoices, a dedicated OCR tool is faster and cheaper. But if you need to summarize 100 complex, hand-drawn architecture diagrams, the Multimodal LLM will save you weeks of custom model training.
Does this mean we can stop pre-processing images entirely? Or do we still need to worry about lighting, rotation, and noise before sending the "vision" request to the model?
I've found Gemini 1.5 Pro's long context window amazing for video. You can upload a 10-minute technical demo and it can pinpoint the exact second a specific feature was mentioned.
Absolutely, Margaret! The video-to-text capabilities are underutilized. It’s making qualitative data analysis for user testing so much faster than manual transcription ever was.
Good question, Christopher. While these models are robust, "garbage in, garbage out" still applies. If a graph is blurry or the text is tiny, the LLM will hallucinate the numbers. I still recommend a basic pre-processing step to increase contrast and normalize orientation. Also, keep an eye on "token usage"—sending high-res images can be significantly more expensive than text. I usually downscale images to the model's native input resolution (like 512x512 or 1024x1024) to save on costs without losing much accuracy.