We are building a RAG-based assistant and I'm curious about the Semantic Kernel memory architecture. How does it handle long-term versus short-term memory, and which vector databases are most compatible when working with large enterprise datasets in Azure?
3 answers
Memory in Semantic Kernel is split into "Volatile" (short-term) and "Persistent" (long-term) storage. For short-term context, the ChatHistory class is your go-to. For long-term retrieval, the SemanticTextMemory allows you to connect to vector stores. In the Azure ecosystem, Azure AI Search is the most seamless integration. It handles the indexing and similarity search, which the Semantic Kernel then consumes to "ground" the LLM's responses. A key tip is to use the TextChunker utility to break down large documents before embedding them, ensuring the LLM doesn't hit token limits.
Regarding the vector stores, have you noticed any latency issues when using Semantic Kernel with Qdrant or Milvus compared to Azure AI Search? I'm trying to decide if we should stick with a managed service or go open-source.
Always remember to clear your volatile memory after a session ends, or the Semantic Kernel might carry over irrelevant context into the next user interaction, leading to confusing replies.
Great point, Cynthia. Implementing a session-based cleanup strategy is vital for maintaining the "relevance" and privacy of the AI agent's responses in a multi-user application.
Patrick, the latency usually stems from the embedding model call rather than the database itself. However, Semantic Kernel provides a unified interface (IMemoryStore), so you can swap them easily. Azure AI Search is often faster for enterprise setups due to the backbone integration with Azure OpenAI, but Milvus is excellent if you need to scale to billions of vectors locally.