I'm transitioning from a custom-built WebSocket server to LiveKit for a customer support voice agent. One of our biggest pain points is "barge-in"—the agent keeps talking for 2 seconds after the user interrupts. Does the LiveKit Agents framework handle the suppression of the audio stream natively, or do I need to manually clear the TTS buffer when VAD (Voice Activity Detection) triggers?
3 answers
The beauty of LiveKit is that it’s WebRTC-based, not just a raw WebSocket. The AgentSession includes a turn_detection parameter that uses a specialized Silero VAD model. When it detects user speech, the framework sends an immediate "cancel" signal to the TTS provider and stops the outgoing audio track in the room. In my experience, this reduces the "awkward overlap" to sub-200ms, which is night and day compared to trying to coordinate this manually over WebSockets.
Are you using the default Silero VAD, or have you tried a more sensitive model for noisy environments?
Check out the AgentSession.generate_reply() docs; it has a flag specifically for handling task cancellation during interruptions.
Good point, Charles. Properly handling the interruption means the LLM needs to know it was cut off so it can resume or pivot in the next turn. LiveKit manages that state remarkably well.
I’m sticking with Silero for now. The key is actually tuning the min_speech_duration in LiveKit. If you set it too low, every cough or keyboard click will kill the agent's response, which is just as frustrating for the user as a late interruption.