I'm trying to summarize some very long legal documents, but Ollama seems to cut off the text after about 4096 tokens. I know the base model supports much more. Is there a command in the Modelfile or a CLI flag that allows me to expand the context limit so I can feed larger prompts into the system?
3 answers
You can change this by creating a custom Modelfile. First, run ollama show --modelfile llama3 to see the current config. Copy that into a new file, and add the line PARAMETER num_ctx 8192 (or whatever size you need). Then, create your new version with ollama create llama3-long -f Modelfile. Keep in mind that increasing the context size significantly increases the VRAM usage. When I bumped my Ollama setup to 16k context, I noticed my RAM usage spiked by about 4GB. If you don't have enough VRAM, the model will either crash or become extremely slow.
Does your hardware actually have enough overhead to handle the kv-cache for those larger context windows without swapping to disk?
You can also set it on the fly in the interactive mode by typing /set parameter num_ctx 8192 during your session.
That's a much faster way to test different limits, Daniel! I didn't realize Ollama allowed parameter overrides directly in the chat interface like that.
I have 64GB of system RAM but only 12GB of VRAM. I've found that if I go over 8k context in Ollama, it starts offloading layers to the CPU, which makes the summarization take forever. It's a tough trade-off.