ray
4.3 KiB
name, description
| name | description |
|---|---|
| gpu-llm-homelab | Evaluate GPUs for llama.cpp LLM inference in self-hosted homelab servers — bandwidth analysis, VRAM sizing, CUDA vs Vulkan, case fit, PSU requirements. |
GPU Evaluation for LLM Inference (Homelab)
How to compare GPUs for llama.cpp inference. The metrics that matter are different from gaming.
The Only Numbers That Matter
- Memory bandwidth (GB/s) — THE bottleneck. Token generation speed = bandwidth ÷ model size. Higher = faster.
- VRAM (GB) — determines what models you can fit. Q4_K_M 7B ≈ 4.7 GB, Q4_K_M 14B ≈ 9 GB, Q4_K_M 32B ≈ 18 GB.
- CUDA support — not optional. Vulkan works but is slower and buggier. AMD ROCm doesn't support consumer RDNA cards. No CUDA = permanently second-class.
- Bus width — the hidden trap. 128-bit cards (RTX 4060/5060 series) cap bandwidth regardless of VRAM. A 16GB card on 128-bit is slower than an 8GB card on 256-bit.
Everything else (CUDA core count, clock speed, architecture generation, Tensor cores) is secondary for llama.cpp.
The 128-Bit Bus Trap
NVIDIA's xx60 series has been on 128-bit since 4060 generation. These cards are gaming-first designs where cache compensates for narrow bus — but LLMs don't benefit from cache. Result: a 4060 Ti 16GB (288 GB/s) is slower than a GTX 1080 from 2016 (320 GB/s) at 4× the price. Always check the bus width before getting excited about VRAM.
AMD Cards for LLMs
AMD cards look good on paper (more VRAM per dollar) but:
- No CUDA — stuck on Vulkan backend
- ROCm excludes consumer RDNA/RDNA2/RDNA3/RDNA4 cards
- llama.cpp Vulkan has fewer optimizations, more bugs
- Only worth it if the card is dramatically cheaper per GB than NVIDIA
Unless the user explicitly wants AMD for gaming or gets an extreme deal, steer toward NVIDIA + CUDA.
Evaluation Checklist
When a user asks about a GPU for LLM:
- Look up specs: bandwidth, VRAM, bus width, CUDA cores (TechPowerUp GPU database)
- Calculate token speed relative to current card: bandwidth_ratio × backend_factor (CUDA ~20% faster than Vulkan)
- Model ceiling: what's the biggest model at Q4_K_M (VRAM × 0.8)?
- Check PSU: wattage headroom (GPU TDP + CPU TDP + 50W), available PCIe power connectors
- Check case: GPU length, slot width, cooling type (blower vs open-air), airflow
- Price-to-bandwidth ratio: $/GB_per_second = price ÷ bandwidth. Lower is better.
Power Consumption Reality
- LLM inference is bursty — GPU idles at 25-35W, spikes to TDP for seconds per request
- 24/7 idle cost dominates. At $0.13/kWh: 25W idle = $2.34/month, 30W idle = $2.81/month
- Difference between cards at idle is pennies. Peak draw only matters for PSU sizing.
- Power limit with
nvidia-smi -plcan reduce noise with near-zero inference speed loss (bandwidth-bound, not core-bound)
Prebuilt Desktop Pitfalls
Prebuilt systems (HP OMEN, Dell XPS, Alienware) have constraints:
- Proprietary PSUs: may be non-standard form factor. Check before assuming ATX fits.
- Limited PCIe cables: prebuilt PSUs often have fewer connectors than aftermarket. A 750W prebuilt PSU might only have 2× 8-pin even though a retail 750W has 4.
- GPU clearance: measure, don't assume. 300mm is a common max for mid-towers.
- Cooling: open-air GPUs dump heat into the case. In airflow-limited prebuilts, blower-style (rear exhaust) cards are safer.
- Power limit as noise control:
sudo nvidia-smi -pm 1 && sudo nvidia-smi -pl 280caps a 3090 at 280W with minimal speed loss.
Value Tiers (used market, mid-2025)
| Tier | Price | Cards | Bandwidth | VRAM |
|---|---|---|---|---|
| Budget | $80-100 | GTX 1080 8GB | 320 GB/s | 8 GB |
| Sweet spot | $150-170 | GTX 1080 Ti 11GB | 484 GB/s | 11 GB |
| Best value | $280-320 | RTX 2080 Ti 11GB | 616 GB/s | 11 GB |
| High-end | $600-700 | RTX 3090 24GB | 936 GB/s | 24 GB |
| Overkill | $1200+ | RTX 4090 24GB | 1008 GB/s | 24 GB |
Skip the 128-bit generation (4060/5060/4060 Ti/5060 Ti) entirely. Skip AMD unless the deal is absurd.
After Purchase
- Enable persistence mode:
sudo nvidia-smi -pm 1 - Set conservative power limit:
sudo nvidia-smi -pl <watts>(80% of TDP is safe) - CUDA backend in llama.cpp:
-ngl 99(offload all layers) - Verify with
nvidia-smi— should show llama-server process with expected VRAM