I have an existing project built with LangChain that calls the GPT-4 API. I want to switch to a local model for privacy. Can I just point the base URL to my Ollama instance? I’m specifically looking for information on whether the chat completion endpoints are fully compatible with the OpenAI SDK format.
3 answers
Yes, this is one of the best features of the platform. Ollama provides an OpenAI-compatible endpoint at http://localhost:11434/v1. In LangChain, you can use the ChatOpenAI class and simply pass the base_url parameter pointing to your local machine. You’ll need to provide a dummy API key (like 'ollama') because the SDK expects one, even though the local server doesn't check it. I successfully migrated a document processing pipeline to this setup in mid-2024, and it saved us a fortune in API costs while keeping all our client data on-premise.
Does the Ollama implementation support function calling and tools in the same way the latest OpenAI models do?
The streaming responses also work out of the box, which is vital for maintaining a good UX in chat applications.
That’s a huge relief. I was worried I’d have to rewrite my frontend logic, but if Ollama handles the SSE stream properly, it should be a seamless transition.
It's getting there, Kenneth! While the basic chat completions work perfectly, complex tool calling depends heavily on the model you are using (like Llama 3.1). Ollama recently added better support for structured outputs, which makes building those agents much more reliable than it was a year ago.