initial commit

skills/mlops/inference/llama-cpp/references/vram-sizing.md

@@ -0,0 +1,63 @@
+# VRAM Budget & Model Sizing Guide
+
+## Formula for Q4_K_M
+
+VRAM ≈ params_in_billions × 0.58 + context_vram (1–3 GB for 8K–32K context)
+
+**Tight fit check:** If a model loads but crashes on the first prompt, it's VRAM-starved. Drop one size tier.
+
+## Model Size Reference
+
+Approximate VRAM at **Q4_K_M** (Ollama default):
+
+| Params | Weight size | Total VRAM | Fits in |
+|--------|-------------|-----------|---------|
+| 1B–3B | 0.6–1.8 GB | 2–3 GB | Any GPU |
+| 7B–8B | 4–5 GB | 5–7 GB | 6 GB+, comfortable on 8 GB |
+| 12B–14B | 7–9 GB | 9–12 GB | 11 GB+ |
+| 22B–24B | 12–14 GB | 14–17 GB | 16 GB+ |
+| 32B–35B | 18–21 GB | 20–24 GB | 24 GB |
+| 70B–72B | 38–42 GB | 42–48 GB | 48 GB+ or dual GPU |
+
+## Quant Upsizing
+
+| Quant | Multiplier | 7B model | 14B model | 34B model |
+|-------|-----------|----------|-----------|-----------|
+| Q2_K | ×0.33 | 2.3 GB | 4.6 GB | 11.2 GB |
+| Q3_K_M | ×0.40 | 2.8 GB | 5.6 GB | 13.6 GB |
+| Q4_K_M | ×0.58 | 4.1 GB | 8.1 GB | 19.7 GB |
+| Q5_K_M | ×0.68 | 4.8 GB | 9.5 GB | 23.1 GB |
+| Q6_K | ×0.80 | 5.6 GB | 11.2 GB | 27.2 GB |
+| Q8_0 | ×1.00 | 7.0 GB | 14.0 GB | 34.0 GB |
+
+## Known-Good GPU Combos
+
+| GPU | VRAM | Best LLM (Q4_K_M) | Best Vision |
+|-----|------|--------------------|-------------|
+| RTX 3060 | 12 GB | qwen2.5:14b or mistral-nemo:12b | llava:13b or llava-llama3:8b |
+| RTX 2080 Ti | 11 GB | qwen2.5:14b (tight) or mistral-nemo:12b | llava:13b (tight) or llava-llama3:8b |
+| RTX 3090 | 24 GB | qwen2.5:32b or llama3:70b (Q3) | llava:34b or llama3.2-vision:11b |
+| RTX 4090 | 24 GB | Same as 3090 | Same |
+| RTX 4070 | 12 GB | Same as 3060 | Same |
+
+## Ollama GPU Investigation
+
+When the user asks "what's using my GPU":
+
+1. `nvidia-smi` — all GPU processes with PIDs and VRAM usage
+2. `ps -p <PID> -o pid,args --no-headers` — which model and port each process runs
+3. `curl -s http://localhost:11434/api/ps | python3 -m json.tool` — Ollama model details, quant, expiry
+4. `journalctl -u ollama --since "5 min ago" --no-pager` — recent Ollama activity
+
+The `expires_at` field tells when Ollama auto-unloads (default 5 min idle).
+
+## Removing Stale llama.cpp Server
+
+```bash
+kill <PID>
+rm -rf /path/to/build /path/to/model.gguf
+sudo systemctl stop llama-server 2>/dev/null
+sudo systemctl disable llama-server 2>/dev/null
+sudo rm /etc/systemd/system/llama-server.service
+sudo systemctl daemon-reload
+```