---
name: gpu-llm-homelab
description: Evaluate GPUs for llama.cpp LLM inference in self-hosted homelab servers — bandwidth analysis, VRAM sizing, CUDA vs Vulkan, case fit, PSU requirements.
---

# GPU Evaluation for LLM Inference (Homelab)

How to compare GPUs for llama.cpp inference. The metrics that matter are different from gaming.

## The Only Numbers That Matter

1. **Memory bandwidth (GB/s)** — THE bottleneck. Token generation speed = bandwidth ÷ model size. Higher = faster.
2. **VRAM (GB)** — determines what models you can fit. Q4_K_M 7B ≈ 4.7 GB, Q4_K_M 14B ≈ 9 GB, Q4_K_M 32B ≈ 18 GB.
3. **CUDA support** — not optional. Vulkan works but is slower and buggier. AMD ROCm doesn't support consumer RDNA cards. No CUDA = permanently second-class.
4. **Bus width** — the hidden trap. 128-bit cards (RTX 4060/5060 series) cap bandwidth regardless of VRAM. A 16GB card on 128-bit is slower than an 8GB card on 256-bit.

Everything else (CUDA core count, clock speed, architecture generation, Tensor cores) is secondary for llama.cpp.

## The 128-Bit Bus Trap

NVIDIA's xx60 series has been on 128-bit since 4060 generation. These cards are gaming-first designs where cache compensates for narrow bus — but LLMs don't benefit from cache. Result: a 4060 Ti 16GB (288 GB/s) is **slower than a GTX 1080 from 2016** (320 GB/s) at 4× the price. Always check the bus width before getting excited about VRAM.

## AMD Cards for LLMs

AMD cards look good on paper (more VRAM per dollar) but:
- No CUDA — stuck on Vulkan backend
- ROCm excludes consumer RDNA/RDNA2/RDNA3/RDNA4 cards
- llama.cpp Vulkan has fewer optimizations, more bugs
- Only worth it if the card is **dramatically** cheaper per GB than NVIDIA

Unless the user explicitly wants AMD for gaming or gets an extreme deal, steer toward NVIDIA + CUDA.

## Evaluation Checklist

When a user asks about a GPU for LLM:

1. **Look up specs**: bandwidth, VRAM, bus width, CUDA cores (TechPowerUp GPU database)
2. **Calculate token speed relative to current card**: bandwidth_ratio × backend_factor (CUDA ~20% faster than Vulkan)
3. **Model ceiling**: what's the biggest model at Q4_K_M (VRAM × 0.8)?
4. **Check PSU**: wattage headroom (GPU TDP + CPU TDP + 50W), available PCIe power connectors
5. **Check case**: GPU length, slot width, cooling type (blower vs open-air), airflow
6. **Price-to-bandwidth ratio**: $/GB_per_second = price ÷ bandwidth. Lower is better.

## Power Consumption Reality

- LLM inference is bursty — GPU idles at 25-35W, spikes to TDP for seconds per request
- 24/7 idle cost dominates. At $0.13/kWh: 25W idle = $2.34/month, 30W idle = $2.81/month
- Difference between cards at idle is pennies. Peak draw only matters for PSU sizing.
- Power limit with `nvidia-smi -pl` can reduce noise with near-zero inference speed loss (bandwidth-bound, not core-bound)

## Prebuilt Desktop Pitfalls

Prebuilt systems (HP OMEN, Dell XPS, Alienware) have constraints:

- **Proprietary PSUs**: may be non-standard form factor. Check before assuming ATX fits.
- **Limited PCIe cables**: prebuilt PSUs often have fewer connectors than aftermarket. A 750W prebuilt PSU might only have 2× 8-pin even though a retail 750W has 4.
- **GPU clearance**: measure, don't assume. 300mm is a common max for mid-towers.
- **Cooling**: open-air GPUs dump heat into the case. In airflow-limited prebuilts, blower-style (rear exhaust) cards are safer.
- **Power limit as noise control**: `sudo nvidia-smi -pm 1 && sudo nvidia-smi -pl 280` caps a 3090 at 280W with minimal speed loss.

## Value Tiers (used market, mid-2025)

| Tier | Price | Cards | Bandwidth | VRAM |
|------|-------|-------|-----------|------|
| Budget | $80-100 | GTX 1080 8GB | 320 GB/s | 8 GB |
| Sweet spot | $150-170 | GTX 1080 Ti 11GB | 484 GB/s | 11 GB |
| Best value | $280-320 | RTX 2080 Ti 11GB | 616 GB/s | 11 GB |
| High-end | $600-700 | RTX 3090 24GB | 936 GB/s | 24 GB |
| Overkill | $1200+ | RTX 4090 24GB | 1008 GB/s | 24 GB |

Skip the 128-bit generation (4060/5060/4060 Ti/5060 Ti) entirely. Skip AMD unless the deal is absurd.

## After Purchase

- Enable persistence mode: `sudo nvidia-smi -pm 1`
- Set conservative power limit: `sudo nvidia-smi -pl <watts>` (80% of TDP is safe)
- CUDA backend in llama.cpp: `-ngl 99` (offload all layers)
- Verify with `nvidia-smi` — should show llama-server process with expected VRAM