While Microsoft emphasizes Azure, I'm working on a project with strict data privacy where we must run everything on-premise. Is it possible to point the Semantic Kernel to a local Oobabooga or Ollama endpoint instead of Azure OpenAI?
3 answers
Yes, absolutely! The Semantic Kernel is provider-agnostic. You can use the HuggingFace connector or, more commonly, use a custom HttpClient to point to a local OpenAI-compatible endpoint. If you are using Ollama, you just need to configure the kernel to use the local URL (usually localhost:11434). This allows you to leverage the orchestration features of the Semantic Kernel—like plugins and planners—while keeping all data processing within your local infrastructure. It’s a great way to balance the power of the SDK with the privacy requirements of an on-premise deployment.
If I switch to a local model, will the Semantic Kernel planners still work efficiently? I’ve heard that smaller models like Llama 3-8B might struggle with the complex reasoning required for sequential planning.
I am currently using this setup with LM Studio. The Semantic Kernel handles the OpenAI-compatible API flawlessly, making it very easy to test different local models.
LM Studio is a fantastic tip, Rebecca. It really simplifies the local setup and lets you focus on the Semantic Kernel logic rather than the infrastructure.
Douglas, that is a valid concern. The "Stepwise Planner" in Semantic Kernel relies heavily on the model's ability to follow complex instructions. While 8B models might fail, the 70B versions usually perform quite well. You might need to refine your "Semantic Function" prompts to be more explicit, as local models aren't always as "forgiving" as GPT-4o when it comes to intent parsing.