For privacy reasons, I cannot send data to OpenAI. I am trying to run Llama 3 locally. How do I configure Spring AI to point to a local Ollama instance? Are there specific dependencies I need, and how does the performance compare when running on a standard developer laptop?
3 answers
Running local models with Ollama is surprisingly simple with Spring AI. You need to add the spring-ai-ollama-spring-boot-starter dependency. In your application.properties, set spring.ai.ollama.base-url to your local Ollama endpoint (usually http://localhost:11434) and specify the model, like spring.ai.ollama.chat.model=llama3. Performance depends heavily on your RAM and GPU. On a Mac with M-series chips, it’s quite snappy. The great thing is that the ChatClient code remains identical to what you’d use for OpenAI, making the swap effortless.
Is there a way to handle multi-modal inputs with Ollama in Spring AI yet? I want to pass images to a local Llava model for analysis within my Spring app.
This is a game changer for dev environments. You can test your entire AI logic offline without worrying about API costs or data leaks during the initial build phase.
Exactly, Diana. It makes the developer inner-loop so much faster. No more waiting for network latency just to test a prompt template change!
Patrick, yes, Spring AI's Ollama implementation supports multimodality. You can use the UserMessage class and attach a Media object containing the image data and its MIME type. When the LLM receives the prompt, it will process the image alongside the text. Just ensure you are running a model that supports vision, like llava or moondream, within your local Ollama environment first.