We’re aiming for a sub-500ms response time. Currently using Deepgram for STT, GPT-4o-mini for the LLM, and Cartesia for TTS, all orchestrated via LiveKit. We are currently at about 850ms. Are there any "pre-warming" techniques or specific LiveKit flags I should use to shave off those last few hundred milliseconds?
3 answers
The biggest win is usually co-location. If your LiveKit agent is running in AWS us-east-1, but your Cartesia or Deepgram requests are jumping across the country, you're losing 100ms right there. Another tip is to use the ctx.wait_for_participant() to pre-warm your STT and TTS connections before the user actually joins. In my setup, pre-warming the Cartesia WebSocket reduced my TTFW by nearly 200ms because the initial handshake was already out of the way.
Are you using the max_tokens limit on your LLM to ensure the first chunk of text is emitted as fast as possible?
Try switching to the deepgram/nova-3 model via LiveKit Inference; it's optimized specifically for these real-time pipelines.
Great tip. Nova-3 has been a game-changer for speed without sacrificing transcription accuracy in noisy environments.
I am, Ryan. I also found that using a smaller system prompt helps. If the LLM has to parse 2000 tokens of "instructions" every time, the time-to-first-token (TTFT) suffers, which ripples down to the LiveKit audio output.