VRAM Budget & Model Sizing Guide

Formula for Q4_K_M

VRAM ≈ params_in_billions × 0.58 + context_vram (1–3 GB for 8K–32K context)

Tight fit check: If a model loads but crashes on the first prompt, it's VRAM-starved. Drop one size tier.

Model Size Reference

Approximate VRAM at Q4_K_M (Ollama default):

Params	Weight size	Total VRAM	Fits in
1B–3B	0.6–1.8 GB	2–3 GB	Any GPU
7B–8B	4–5 GB	5–7 GB	6 GB+, comfortable on 8 GB
12B–14B	7–9 GB	9–12 GB	11 GB+
22B–24B	12–14 GB	14–17 GB	16 GB+
32B–35B	18–21 GB	20–24 GB	24 GB
70B–72B	38–42 GB	42–48 GB	48 GB+ or dual GPU

Quant Upsizing

Quant	Multiplier	7B model	14B model	34B model
Q2_K	×0.33	2.3 GB	4.6 GB	11.2 GB
Q3_K_M	×0.40	2.8 GB	5.6 GB	13.6 GB
Q4_K_M	×0.58	4.1 GB	8.1 GB	19.7 GB
Q5_K_M	×0.68	4.8 GB	9.5 GB	23.1 GB
Q6_K	×0.80	5.6 GB	11.2 GB	27.2 GB
Q8_0	×1.00	7.0 GB	14.0 GB	34.0 GB

Known-Good GPU Combos

GPU	VRAM	Best LLM (Q4_K_M)	Best Vision
RTX 3060	12 GB	qwen2.5:14b or mistral-nemo:12b	llava:13b or llava-llama3:8b
RTX 2080 Ti	11 GB	qwen2.5:14b (tight) or mistral-nemo:12b	llava:13b (tight) or llava-llama3:8b
RTX 3090	24 GB	qwen2.5:32b or llama3:70b (Q3)	llava:34b or llama3.2-vision:11b
RTX 4090	24 GB	Same as 3090	Same
RTX 4070	12 GB	Same as 3060	Same

Ollama GPU Investigation

When the user asks "what's using my GPU":

nvidia-smi — all GPU processes with PIDs and VRAM usage
ps -p <PID> -o pid,args --no-headers — which model and port each process runs
curl -s http://localhost:11434/api/ps | python3 -m json.tool — Ollama model details, quant, expiry
journalctl -u ollama --since "5 min ago" --no-pager — recent Ollama activity

The expires_at field tells when Ollama auto-unloads (default 5 min idle).

Removing Stale llama.cpp Server

kill <PID>
rm -rf /path/to/build /path/to/model.gguf
sudo systemctl stop llama-server 2>/dev/null
sudo systemctl disable llama-server 2>/dev/null
sudo rm /etc/systemd/system/llama-server.service
sudo systemctl daemon-reload

2.4 KiB Raw Permalink Blame History Unescape Escape