I'm working on a project that involves AI and Deep Learning where we need to display extracted data to the user as it's being generated. I know raw streaming returns chunks of text, but how does the Instructor library handle streaming partial JSON objects? Is it possible to maintain the Pydantic schema while the model is still "typing"?
3 answers
Streaming partial objects is actually one of the coolest features here. Instructor provides a Partial type that allows you to yield objects that are only partially filled. As the LLM sends more tokens, the object updates. This is incredibly useful for UI/UX in deep learning applications because you can show the user the name of an entity the moment it's identified, even while the rest of the attributes are still being processed. It keeps the application feeling responsive without sacrificing the structure you need for your backend.
Is there a risk of the UI crashing if the partial object doesn't meet the full validation yet?
I use this for a real-time transcription tool. Seeing the data structure build up live is much better than waiting for the final block.
Agreed, Daniel. It makes the "black box" of LLM generation much more transparent and useful for end-users.
Good question, Patrick! The library handles this by making all fields optional in the partial model during the stream. It doesn't throw a validation error until the very final completion if a required field is still missing. So, your UI just needs to handle null or empty values gracefully while the data is populating. It’s quite stable for frontend consumption once you set up the state management.