I want to build an agent that can "see" what the user is showing via their smartphone camera while talking to them. I see that LiveKit recently added support for multimodal models like Gemini 2.0. Is it possible to feed the video track from a LiveKit room directly into Gemini's vision input in real-time, or do I have to take snapshots every few seconds?
3 answers
You can actually stream the video track directly. In the LiveKit Python SDK, you can subscribe the agent to the user's video track and use a VideoStream to capture frames. For Gemini Live, you typically use their Realtime API which accepts a media stream. The latency is impressively low—around 300-500ms for visual recognition. This is much better than the "snapshot" approach because the agent can react to movement, like "I see you're holding a red screwdriver," without the user having to stop and wait for an upload.
Are you concerned about the bandwidth costs of streaming high-res video to an AI agent for a 10-minute session?
Check out the "Aura" visualizer in the new Agents UI library—it makes the multimodal experience feel much more polished on the frontend.
Thanks Charles! Having a UI that actually reacts when the agent "sees" something is a huge boost for user engagement.
Definitely a concern. I usually downscale the track to 720p at 15fps before sending it to the agent. Most vision models don't need 4K 60fps to identify objects, and it keeps the LiveKit egress costs manageable.