I just installed Ollama on my Linux workstation, but it seems to be defaulting to my integrated graphics rather than my RTX 3090. Is there a specific environment variable I need to set so that Ollama recognizes the discrete GPU? I want to run the Llama 3.1 70B model, and right now the token generation speed is extremely slow because it's falling back to the CPU.
3 answers
To fix this, you need to use the CUDA_VISIBLE_DEVICES environment variable. By default, Ollama tries to auto-detect the best available hardware, but sometimes it gets confused by multi-GPU setups. You should run nvidia-smi to find the index of your RTX 3090 (usually 0 or 1). Then, stop the service and restart it using CUDA_VISIBLE_DEVICES=0 ollama serve. This forces the application to ignore other devices. I had a similar issue with my dual-GPU rig last year, and once I pinned the service to the correct ID, my tokens-per-second jumped from 2 to nearly 50.
Are you running this inside a Docker container, or is it a bare-metal installation on your Ubuntu or Fedora distribution?
Check your logs in /var/log/ollama. It will explicitly tell you if it failed to initialize the CUDA library.
Good call, Brandon. Reading the logs is the fastest way to see if Ollama is failing because of a library mismatch or a VRAM allocation error during the initial model load.
I'm running it bare-metal on Ubuntu 22.04. I noticed that if the NVIDIA Container Toolkit isn't perfectly configured, Ollama sometimes struggles to bind to the drivers even outside of Docker. Reinstalling the latest 545 drivers usually clears up the communication path between the hardware and the CLI.