I'm building a virtual companion using <LiveKit (AI voice)>. How can I ensure the agent remembers what the user said in a previous call? Since each session is technically a new room, I'm struggling with where to store the conversation summaries and how to inject them back into the agent's context when the same user reconnects a week later.
3 answers
The best way to handle persistence with <LiveKit (AI voice)> is by using a combination of a metadata database and the 'JobContext'. When a user joins, use their unique identifier to fetch their profile from a database like MongoDB or Redis. You can then pass this historical data into the 'AgentSession' as part of the initial instructions. For saving data, I recommend using the 'on_session_finished' event to trigger a summarization step where the LLM extracts key facts and saves them back to your DB for the next encounter.
Would it be more efficient to use a Vector DB for RAG-based memory instead of just a summary?
Using 'context_variables' is definitely the way to go to keep things organized.
I’ve found that even a simple JSON blob of "Facts about the user" injected into the
It depends on the scale, Scott. For a companion, a RAG setup is great because the agent can call a tool to "search memory" only when needed. This keeps the prompt short and saves tokens, while still allowing the agent to recall specific details from months ago if the user brings them up.