Our project requires the <LiveKit (AI voice)> agent to automatically detect and switch between English and Spanish based on the user's input. Does the current SDK support dynamic model swapping mid-session, or do we need to initialize separate agents for each language? I'm worried about the memory overhead on our server if we have to run multiple inference pipelines.
3 answers
You don't necessarily need separate agents for the <LiveKit (AI voice)> workflow. Most modern STT providers like Deepgram or OpenAI's Whisper (integrated via the LiveKit plugins) support 'language="multi"' or auto-detection. When the STT engine identifies a language change, it passes the transcript to the LLM, which is naturally polyglot. The trick is to update the TTS 'voice' property dynamically. You can use the 'on_speech_detected' hook to trigger a logic check and swap the output voice ID so the agent responds in the correct dialect.
If the user speaks Spanglish, does the agent get confused between the two active TTS profiles?
The memory overhead is actually quite low since the heavy lifting is handled by the model providers, not your host.
Exactly, the
It can be tricky, Tyler. In those cases, I usually stick to a single multi-lingual voice model (like those from Cartesia or ElevenLabs) that handles both languages natively without needing a hard swap, which keeps the experience much smoother.