# VRAM Budget & Model Sizing Guide

## Formula for Q4_K_M

VRAM ≈ params_in_billions × 0.58 + context_vram (1–3 GB for 8K–32K context)

**Tight fit check:** If a model loads but crashes on the first prompt, it's VRAM-starved. Drop one size tier.

## Model Size Reference

Approximate VRAM at **Q4_K_M** (Ollama default):

| Params | Weight size | Total VRAM | Fits in |
|--------|-------------|-----------|---------|
| 1B–3B | 0.6–1.8 GB | 2–3 GB | Any GPU |
| 7B–8B | 4–5 GB | 5–7 GB | 6 GB+, comfortable on 8 GB |
| 12B–14B | 7–9 GB | 9–12 GB | 11 GB+ |
| 22B–24B | 12–14 GB | 14–17 GB | 16 GB+ |
| 32B–35B | 18–21 GB | 20–24 GB | 24 GB |
| 70B–72B | 38–42 GB | 42–48 GB | 48 GB+ or dual GPU |

## Quant Upsizing

| Quant | Multiplier | 7B model | 14B model | 34B model |
|-------|-----------|----------|-----------|-----------|
| Q2_K | ×0.33 | 2.3 GB | 4.6 GB | 11.2 GB |
| Q3_K_M | ×0.40 | 2.8 GB | 5.6 GB | 13.6 GB |
| Q4_K_M | ×0.58 | 4.1 GB | 8.1 GB | 19.7 GB |
| Q5_K_M | ×0.68 | 4.8 GB | 9.5 GB | 23.1 GB |
| Q6_K | ×0.80 | 5.6 GB | 11.2 GB | 27.2 GB |
| Q8_0 | ×1.00 | 7.0 GB | 14.0 GB | 34.0 GB |

## Known-Good GPU Combos

| GPU | VRAM | Best LLM (Q4_K_M) | Best Vision |
|-----|------|--------------------|-------------|
| RTX 3060 | 12 GB | qwen2.5:14b or mistral-nemo:12b | llava:13b or llava-llama3:8b |
| RTX 2080 Ti | 11 GB | qwen2.5:14b (tight) or mistral-nemo:12b | llava:13b (tight) or llava-llama3:8b |
| RTX 3090 | 24 GB | qwen2.5:32b or llama3:70b (Q3) | llava:34b or llama3.2-vision:11b |
| RTX 4090 | 24 GB | Same as 3090 | Same |
| RTX 4070 | 12 GB | Same as 3060 | Same |

## Ollama GPU Investigation

When the user asks "what's using my GPU":

1. `nvidia-smi` — all GPU processes with PIDs and VRAM usage
2. `ps -p <PID> -o pid,args --no-headers` — which model and port each process runs
3. `curl -s http://localhost:11434/api/ps | python3 -m json.tool` — Ollama model details, quant, expiry
4. `journalctl -u ollama --since "5 min ago" --no-pager` — recent Ollama activity

The `expires_at` field tells when Ollama auto-unloads (default 5 min idle).

## Removing Stale llama.cpp Server

```bash
kill <PID>
rm -rf /path/to/build /path/to/model.gguf
sudo systemctl stop llama-server 2>/dev/null
sudo systemctl disable llama-server 2>/dev/null
sudo rm /etc/systemd/system/llama-server.service
sudo systemctl daemon-reload
```