ray
64 lines
2.4 KiB
Markdown
64 lines
2.4 KiB
Markdown
# VRAM Budget & Model Sizing Guide
|
||
|
||
## Formula for Q4_K_M
|
||
|
||
VRAM ≈ params_in_billions × 0.58 + context_vram (1–3 GB for 8K–32K context)
|
||
|
||
**Tight fit check:** If a model loads but crashes on the first prompt, it's VRAM-starved. Drop one size tier.
|
||
|
||
## Model Size Reference
|
||
|
||
Approximate VRAM at **Q4_K_M** (Ollama default):
|
||
|
||
| Params | Weight size | Total VRAM | Fits in |
|
||
|--------|-------------|-----------|---------|
|
||
| 1B–3B | 0.6–1.8 GB | 2–3 GB | Any GPU |
|
||
| 7B–8B | 4–5 GB | 5–7 GB | 6 GB+, comfortable on 8 GB |
|
||
| 12B–14B | 7–9 GB | 9–12 GB | 11 GB+ |
|
||
| 22B–24B | 12–14 GB | 14–17 GB | 16 GB+ |
|
||
| 32B–35B | 18–21 GB | 20–24 GB | 24 GB |
|
||
| 70B–72B | 38–42 GB | 42–48 GB | 48 GB+ or dual GPU |
|
||
|
||
## Quant Upsizing
|
||
|
||
| Quant | Multiplier | 7B model | 14B model | 34B model |
|
||
|-------|-----------|----------|-----------|-----------|
|
||
| Q2_K | ×0.33 | 2.3 GB | 4.6 GB | 11.2 GB |
|
||
| Q3_K_M | ×0.40 | 2.8 GB | 5.6 GB | 13.6 GB |
|
||
| Q4_K_M | ×0.58 | 4.1 GB | 8.1 GB | 19.7 GB |
|
||
| Q5_K_M | ×0.68 | 4.8 GB | 9.5 GB | 23.1 GB |
|
||
| Q6_K | ×0.80 | 5.6 GB | 11.2 GB | 27.2 GB |
|
||
| Q8_0 | ×1.00 | 7.0 GB | 14.0 GB | 34.0 GB |
|
||
|
||
## Known-Good GPU Combos
|
||
|
||
| GPU | VRAM | Best LLM (Q4_K_M) | Best Vision |
|
||
|-----|------|--------------------|-------------|
|
||
| RTX 3060 | 12 GB | qwen2.5:14b or mistral-nemo:12b | llava:13b or llava-llama3:8b |
|
||
| RTX 2080 Ti | 11 GB | qwen2.5:14b (tight) or mistral-nemo:12b | llava:13b (tight) or llava-llama3:8b |
|
||
| RTX 3090 | 24 GB | qwen2.5:32b or llama3:70b (Q3) | llava:34b or llama3.2-vision:11b |
|
||
| RTX 4090 | 24 GB | Same as 3090 | Same |
|
||
| RTX 4070 | 12 GB | Same as 3060 | Same |
|
||
|
||
## Ollama GPU Investigation
|
||
|
||
When the user asks "what's using my GPU":
|
||
|
||
1. `nvidia-smi` — all GPU processes with PIDs and VRAM usage
|
||
2. `ps -p <PID> -o pid,args --no-headers` — which model and port each process runs
|
||
3. `curl -s http://localhost:11434/api/ps | python3 -m json.tool` — Ollama model details, quant, expiry
|
||
4. `journalctl -u ollama --since "5 min ago" --no-pager` — recent Ollama activity
|
||
|
||
The `expires_at` field tells when Ollama auto-unloads (default 5 min idle).
|
||
|
||
## Removing Stale llama.cpp Server
|
||
|
||
```bash
|
||
kill <PID>
|
||
rm -rf /path/to/build /path/to/model.gguf
|
||
sudo systemctl stop llama-server 2>/dev/null
|
||
sudo systemctl disable llama-server 2>/dev/null
|
||
sudo rm /etc/systemd/system/llama-server.service
|
||
sudo systemctl daemon-reload
|
||
```
|