How to size RAM for Ollama models on a CPU-only server

Start from the download size

A guide published on dev.to by VeloxaHost, a CPU server hosting company, walks self-hosters through picking Ollama models that fit CPU-only machines. The method: start from the model's download size, mind the quantization defaults, cap the model at roughly half of system RAM, then verify on the server itself.

According to the guide, the size listed for each tag in Ollama's model library is a good proxy for how much memory the model occupies once loaded, before any context overhead is added. Sizes range widely: gemma3:270m is 292 MB, llama3:2:1b is 1.3 GB, llama3.2:3b is 2.0 GB, mistral:7b is 4.4 GB, llama3.1:8b is 4.9 GB, gemma3:12b is 8.2 GB, and gemma3:27b tops the list at 17 GB.

Quantization defaults vary between models

Many tags default to 4-bit quantization, but not all of them do, and the gap is large enough to matter. The guide notes that llama3.2:1b ships as 8-bit by default at 1.3 GB, while its 4-bit variant is only 808 MB. Going the other way, llama3.1:8b is 4.9 GB at 4-bit but 8.5 GB at 8-bit. The recommendation is to pull the fully specified tag when precision matters, for example llama3.2:1b-instruct-q4_K_M.

The rule of thumb: half your RAM

The guide's central heuristic — which its author stresses is their own rule, not an Ollama recommendation — is to keep the model file at no more than about half of the server's RAM at default context, then confirm empirically. Applied across common server sizes:

RAM	What fits comfortably
2 GB	~1 GB models: gemma3:1b, qwen3:0.6b, llama3.2:1b (4-bit)
4 GB	1B–3B models up to ~2 GB: llama3.2:3b, qwen3:1.7b
8 GB	4B models (gemma3:4b, qwen3:4b, phi4-mini); 7B–8B with less headroom
16 GB	8B with longer context; 12B–14B such as gemma3:12b, qwen3:14b

A 27B model, the guide adds, needs more than 16 GB — and on CPU, larger also means slower, so memory is not the only constraint.

Four settings that add memory on top

Beyond the model file, four environment variables determine total memory use. OLLAMA_CONTEXT_LENGTH sets the default context; Ollama falls back to 4096 tokens when there is no large GPU, and longer contexts cost more memory. OLLAMA_NUM_PARALLEL controls how many requests run concurrently per model (default 1), and the guide cites Ollama's FAQ that RAM scales with the product of parallelism and context length. OLLAMA_MAX_LOADED_MODELS, which defaults to 3 on CPU, should be dropped to 1 on small servers. Finally, OLLAMA_KV_CACHE_TYPE=q8_0 cuts the KV cache to roughly half that of the default f16 with very little precision loss, but it only takes effect when Flash Attention is enabled via OLLAMA_FLASH_ATTENTION=1. All of these are set through systemctl edit ollama.service followed by a restart.

Verify, then measure tokens per second

The verification flow is simple: load a model with an empty prompt using ollama run, then inspect ollama ps — where SIZE shows memory in use, PROCESSOR should read 100% CPU, and CONTEXT confirms what was loaded — and check free -h to confirm the available column still has headroom. Speed comes from the API, which returns eval_count and eval_duration; dividing tokens by time yields tokens per second. Run it a few times, and if it is too slow for the workload, move to a smaller model or add vCPUs.

The troubleshooting section covers common failure modes. A service that restarts under load usually means out-of-memory, so check journalctl. A slow first answer after idle typically means the model unloaded, since the default keep-alive is 5 minutes; raise it with OLLAMA_KEEP_ALIVE=30m. Conversations that forget their beginning indicate the prompt outgrew the context window. And pulled tags accumulate on disk, so unused ones should be removed with ollama rm.

When CPU-only is worth it

Per the guide, CPU inference makes sense for small models, batch jobs and private internal tools, and often does not for high-volume chat. Citing its own self-hosting versus SaaS cost comparison, a small hosted API model worked out cheaper below roughly 150 million input plus 30 million output tokens per month. It is worth noting the author's disclosure: VeloxaHost sells CPU servers, so it has an interest in making CPU hosting approachable — though the sizing method itself is vendor-neutral.

Why it matters

Most LLM deployment guidance assumes a GPU. For self-hosters on small VPS instances, RAM is the real budget, and mistakes are easy to make: a quantization default that doubles a model's footprint, a context setting that quietly eats a gigabyte, or three loaded models where one would do. This cheat sheet turns sizing from guesswork into a repeatable check — pick a model under half your RAM, tune four environment variables, then verify with ollama ps and a tokens-per-second measurement before committing to hardware.