ray
79 lines
4.3 KiB
Markdown
79 lines
4.3 KiB
Markdown
---
|
||
name: gpu-llm-homelab
|
||
description: Evaluate GPUs for llama.cpp LLM inference in self-hosted homelab servers — bandwidth analysis, VRAM sizing, CUDA vs Vulkan, case fit, PSU requirements.
|
||
---
|
||
|
||
# GPU Evaluation for LLM Inference (Homelab)
|
||
|
||
How to compare GPUs for llama.cpp inference. The metrics that matter are different from gaming.
|
||
|
||
## The Only Numbers That Matter
|
||
|
||
1. **Memory bandwidth (GB/s)** — THE bottleneck. Token generation speed = bandwidth ÷ model size. Higher = faster.
|
||
2. **VRAM (GB)** — determines what models you can fit. Q4_K_M 7B ≈ 4.7 GB, Q4_K_M 14B ≈ 9 GB, Q4_K_M 32B ≈ 18 GB.
|
||
3. **CUDA support** — not optional. Vulkan works but is slower and buggier. AMD ROCm doesn't support consumer RDNA cards. No CUDA = permanently second-class.
|
||
4. **Bus width** — the hidden trap. 128-bit cards (RTX 4060/5060 series) cap bandwidth regardless of VRAM. A 16GB card on 128-bit is slower than an 8GB card on 256-bit.
|
||
|
||
Everything else (CUDA core count, clock speed, architecture generation, Tensor cores) is secondary for llama.cpp.
|
||
|
||
## The 128-Bit Bus Trap
|
||
|
||
NVIDIA's xx60 series has been on 128-bit since 4060 generation. These cards are gaming-first designs where cache compensates for narrow bus — but LLMs don't benefit from cache. Result: a 4060 Ti 16GB (288 GB/s) is **slower than a GTX 1080 from 2016** (320 GB/s) at 4× the price. Always check the bus width before getting excited about VRAM.
|
||
|
||
## AMD Cards for LLMs
|
||
|
||
AMD cards look good on paper (more VRAM per dollar) but:
|
||
- No CUDA — stuck on Vulkan backend
|
||
- ROCm excludes consumer RDNA/RDNA2/RDNA3/RDNA4 cards
|
||
- llama.cpp Vulkan has fewer optimizations, more bugs
|
||
- Only worth it if the card is **dramatically** cheaper per GB than NVIDIA
|
||
|
||
Unless the user explicitly wants AMD for gaming or gets an extreme deal, steer toward NVIDIA + CUDA.
|
||
|
||
## Evaluation Checklist
|
||
|
||
When a user asks about a GPU for LLM:
|
||
|
||
1. **Look up specs**: bandwidth, VRAM, bus width, CUDA cores (TechPowerUp GPU database)
|
||
2. **Calculate token speed relative to current card**: bandwidth_ratio × backend_factor (CUDA ~20% faster than Vulkan)
|
||
3. **Model ceiling**: what's the biggest model at Q4_K_M (VRAM × 0.8)?
|
||
4. **Check PSU**: wattage headroom (GPU TDP + CPU TDP + 50W), available PCIe power connectors
|
||
5. **Check case**: GPU length, slot width, cooling type (blower vs open-air), airflow
|
||
6. **Price-to-bandwidth ratio**: $/GB_per_second = price ÷ bandwidth. Lower is better.
|
||
|
||
## Power Consumption Reality
|
||
|
||
- LLM inference is bursty — GPU idles at 25-35W, spikes to TDP for seconds per request
|
||
- 24/7 idle cost dominates. At $0.13/kWh: 25W idle = $2.34/month, 30W idle = $2.81/month
|
||
- Difference between cards at idle is pennies. Peak draw only matters for PSU sizing.
|
||
- Power limit with `nvidia-smi -pl` can reduce noise with near-zero inference speed loss (bandwidth-bound, not core-bound)
|
||
|
||
## Prebuilt Desktop Pitfalls
|
||
|
||
Prebuilt systems (HP OMEN, Dell XPS, Alienware) have constraints:
|
||
|
||
- **Proprietary PSUs**: may be non-standard form factor. Check before assuming ATX fits.
|
||
- **Limited PCIe cables**: prebuilt PSUs often have fewer connectors than aftermarket. A 750W prebuilt PSU might only have 2× 8-pin even though a retail 750W has 4.
|
||
- **GPU clearance**: measure, don't assume. 300mm is a common max for mid-towers.
|
||
- **Cooling**: open-air GPUs dump heat into the case. In airflow-limited prebuilts, blower-style (rear exhaust) cards are safer.
|
||
- **Power limit as noise control**: `sudo nvidia-smi -pm 1 && sudo nvidia-smi -pl 280` caps a 3090 at 280W with minimal speed loss.
|
||
|
||
## Value Tiers (used market, mid-2025)
|
||
|
||
| Tier | Price | Cards | Bandwidth | VRAM |
|
||
|------|-------|-------|-----------|------|
|
||
| Budget | $80-100 | GTX 1080 8GB | 320 GB/s | 8 GB |
|
||
| Sweet spot | $150-170 | GTX 1080 Ti 11GB | 484 GB/s | 11 GB |
|
||
| Best value | $280-320 | RTX 2080 Ti 11GB | 616 GB/s | 11 GB |
|
||
| High-end | $600-700 | RTX 3090 24GB | 936 GB/s | 24 GB |
|
||
| Overkill | $1200+ | RTX 4090 24GB | 1008 GB/s | 24 GB |
|
||
|
||
Skip the 128-bit generation (4060/5060/4060 Ti/5060 Ti) entirely. Skip AMD unless the deal is absurd.
|
||
|
||
## After Purchase
|
||
|
||
- Enable persistence mode: `sudo nvidia-smi -pm 1`
|
||
- Set conservative power limit: `sudo nvidia-smi -pl <watts>` (80% of TDP is safe)
|
||
- CUDA backend in llama.cpp: `-ngl 99` (offload all layers)
|
||
- Verify with `nvidia-smi` — should show llama-server process with expected VRAM
|