We are working on a Software Development project that will last 18 months. We need our AI to remember architectural decisions made in month one. How does Microsoft Semantic Kernel handle this long-term memory without blowing up the context window or token costs?
3 answers
The "Semantic Memory" feature in Microsoft Semantic Kernel is specifically designed for this "needle in a haystack" problem. Instead of stuffing every document into the prompt, the Kernel uses a vector database (like Azure AI Search or Qdrant) to index your project documentation. When you ask a question about month-one decisions, the Kernel performs a similarity search, retrieves only the relevant snippets, and injects them into the current prompt. This keeps your token costs low while ensuring the agent has the exact context it needs. For our 18-month projects, we index every ADR (Architecture Decision Record) as it’s written. The AI then acts as a "Living Documentation" assistant that knows exactly why a specific database schema was chosen a year ago.
Does the agent automatically update the memory as the project evolves, or do we have to manually re-index the vector DB every time a decision changes?
The memory connector is model-agnostic too. You can use one model for the embeddings and another for the chat, which saves a lot on latency.
Good point, Thomas. Using a small embedding model like Ada and then GPT-4 for the reasoning is the most efficient way to scale.
You can automate the indexing, Kevin! We set up a GitHub Action that triggers every time an ADR is merged into the main branch. The action calls a Microsoft Semantic Kernel function that "upserts" the new document into the vector store. This ensures the agent's memory is always in sync with the latest code and documentation. It turns the AI into a real-time expert on your evolving codebase.