Ollama keep_alive bug can silently cancel a pending model swap and stall other requests

The bug

A post on dev.to documents a scheduling flaw in Ollama, the widely used tool for running large language models locally. On a server limited to one loaded model at a time, a request for a different model makes Ollama plan to evict the resident model as soon as it finishes its current work, and the new request waits for that eviction. The flaw: if another request for the resident model arrives during that wait and carries a keep_alive value, the value replaces the pending unload. Instead of evicting the model, Ollama starts an ordinary keep-alive timer, and the queued request waits out the entire keep_alive — silently, with no error anywhere.

The author reproduced this on Ollama 0.40.8 running CPU-only in a disposable LXC container, and it is tracked upstream as ollama/ollama#19182 with a fix proposed in pull request #19190. According to the post, that fix was still open at the time of writing, and no commit message between v0.40.8 and v0.40.9-rc1 mentions it.

The reproduction

The test used two small models — qwen3:0.6b loaded, gemma3:270m waiting — with OLLAMA_MAX_LOADED_MODELS=1, OLLAMA_NUM_PARALLEL=4 and OLLAMA_KEEP_ALIVE=30m. Three requests fired in sequence: a long generation on the loaded model, three seconds later a short request for gemma3:270m, and two seconds after that a one-token request for the loaded model.

The sequence ran twice, once with the third request plain and once with "keep_alive": "30m" attached. The gap was dramatic. Without keep_alive, the gemma3 request returned HTTP 200 after roughly 9.2 seconds, and gemma3 was the loaded model at the end. With the 30-minute value, curl exhausted its 150-second timeout and gave up, and qwen3 was still resident.

Debug logging, enabled via OLLAMA_DEBUG=1, exposes the mechanism. The scheduler logs that the maximum number of runners has been reached and that one will be unloaded to make room, resets the resident model to expire immediately, and waits for pending requests to finish. But the third request causes the resident runner to be re-evaluated, and when it goes idle it is logged as having a non-zero duration, so a timer is added rather than an unload. In the healthy run, the runner expires at zero duration and the unload completes.

Why it stays hidden

From every angle an operator would normally check, the server looks healthy, the post explains. The /api/ps endpoint shows the loaded model, requests to it succeed, and the scheduler messages that reveal the problem appear only at debug level. Nothing errors — the victim request just hangs until the client's own timeout or the keep_alive elapses.

The trigger is also ordinary client behaviour. Many clients and UIs attach keep_alive to every request; according to the post, the reporter lost two hours to a client sending "2h" and thirteen minutes to a health probe. Single-slot setups are standard on machines whose memory holds one model, which the author notes describes most homelab GPUs. Timing matters too: under light use the collision may never happen, while a steady stream of requests for the loaded model makes it recur, so the other model effectively never loads.

What happens inside the scheduler

According to the report's reading of server/sched.go, the flow is as follows. With the single slot full, Ollama picks the resident runner for eviction, zeroes its remaining lifetime so it unloads once its last request completes, and queues the new request. A request for the same model that lands in between is served by the already-loaded runner, and its keep_alive overwrites the runner's session duration. When the runner idles, it is no longer flagged to expire, so it receives a standard keep-alive timer and stays resident, with the other request still queued behind it.

Workarounds until a fix ships

Sending keep_alive: 0 for the loaded model forces the unload; in the test this released the waiting request within 1.5 seconds, after which gemma3 loaded. Other mitigations are to stop clients sending keep_alive per request and set it once server-side with OLLAMA_KEEP_ALIVE, or to allow more than one model slot if memory permits so no eviction is needed — though the author notes the three-request sequence was not rerun with two slots. The post also ships a detection script that reads OLLAMA_MAX_LOADED_MODELS and, when debug logging is on, counts evictions that were converted into keep-alive timers.

Why it matters

This is a quiet failure mode in a popular local-inference server: one client's harmless-looking option overrides the server's own scheduling plan, and a different client pays for it with an indefinite hang. Anyone running single-slot Ollama behind chatty clients or health probes can hit it without ever seeing an error. The broader lesson generalises: when a per-request option and a scheduler write to the same field, ask which one wins — and when something hangs without an error, rerun the scenario with and without the suspect field. Here a single field was the difference between nine seconds and never.