ray
126 lines
8.5 KiB
Markdown
126 lines
8.5 KiB
Markdown
# GPU Hardware Recommendations for llama.cpp
|
||
|
||
Hardware analysis for selecting a GPU for local llama.cpp inference. All analysis assumes llama.cpp as the inference engine.
|
||
|
||
## The Golden Rule: CUDA is Everything
|
||
|
||
For llama.cpp inference, **NVIDIA CUDA is the only sane choice for consumer GPUs**. AMD consumer cards (RDNA2/RDNA3) are locked to Vulkan backend — no ROCm support on consumer SKUs. The Vulkan backend works but has real limitations:
|
||
|
||
- No flash attention
|
||
- No MMQ kernels (slower quants)
|
||
- More bugs, less optimization priority from the llama.cpp team
|
||
- Real-world token generation is often on par with or slower than a weaker NVIDIA card on CUDA, despite higher raw bandwidth
|
||
|
||
**AMD integrated GPUs and iGPUs**: Vulkan works fine for these (they're small anyway). The problem is discrete AMD consumer cards where you're paying for bandwidth you can't fully use.
|
||
|
||
## The 128-Bit Bus Trap
|
||
|
||
NVIDIA's xx60 series (4060, 4060 Ti, 5060, 5060 Ti) uses a crippled **128-bit memory bus** — half the width of older cards like the GTX 1080 (256-bit) or RTX 2080 Ti (352-bit). Even with GDDR6X/GDDR7 speeds, the narrow bus caps effective bandwidth so severely that a $400 RTX 4060 Ti 16GB (288 GB/s) is **slower than an $80 GTX 1080 (320 GB/s)** for LLM token generation.
|
||
|
||
**The 16GB dead zone**: the first NVIDIA 16GB card with a bus wide enough for LLMs is the RTX 4070 Ti Super (256-bit, 672 GB/s) at ~$600. Every 16GB card below it is a 128-bit gaming card masquerading as an AI card — attractive VRAM number, useless bandwidth.
|
||
|
||
## Comparison Table
|
||
|
||
| GPU | VRAM | Bandwidth | Bus | CUDA | Price (used) | Best model fits | Notes |
|
||
|---|---|---|---|---|---|---|---|
|
||
| GTX 1050 Ti | 4 GB | 112 GB/s | 128-bit | ❌ Vulkan only | — | Q2_K 7B | Slot-powered. Baseline. |
|
||
| GTX 1070 | 8 GB | 256 GB/s | 256-bit | ✅ | ~$80-100 | Q4_K_M 7B, Q2 14B | 1× 8-pin, 150W. |
|
||
| **GTX 1080** | 8 GB | 320 GB/s | 256-bit | ✅ | **~$80** | Q5_K_M 7B | Best budget LLM card. Faster than 4060 Ti 16GB. |
|
||
| GTX 1080 Ti | 11 GB | 484 GB/s | 352-bit | ✅ | ~$140-170 | Q4_K_M 14B | Sweet spot. 11GB + 484 GB/s. |
|
||
| **RTX 2080 Ti** | 11 GB | 616 GB/s | 352-bit | ✅ + Tensor | ~$250-300 | Q4_K_M 14B, IQ3 20B | Best sub-$300 card. 2× RTX 2060 12GB speed. |
|
||
| RTX 2060 12GB | 12 GB | 336 GB/s | 192-bit | ✅ + Tensor | ~$200 | Q4_K_M 14B | 1GB more than 1080 Ti but 44% slower. VLMs need 12GB+. |
|
||
| RTX 2060 SUPER | 8 GB | 448 GB/s | 256-bit | ✅ + Tensor | ~$180-220 | Q5_K_M 7B | Fast 7B card, hard-capped at 8GB. |
|
||
| RTX 3070 Ti | 8 GB | 608 GB/s | 256-bit | ✅ + Tensor | ~$250-300 | Q5_K_M 7B | Very fast, but 8GB only — same ceiling as $80 GTX 1080. |
|
||
| RTX 3080 10GB | 10 GB | 760 GB/s | 320-bit | ✅ + Tensor | ~$350-400 | Q4_K_M 14B | Faster than 2080 Ti but 1GB less VRAM. |
|
||
| RTX 3080 12GB | 12 GB | 912 GB/s | 384-bit | ✅ + Tensor | ~$350-400 | Q4_K_M 14B+ | True upgrade from 2080 Ti. 48% faster. |
|
||
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s | **128-bit** ❌ | ✅ + Tensor | $400+ | Q4_K_M 14B | **TRAP.** Slower than $80 GTX 1080. |
|
||
| RTX 5060 Ti 16GB | 16 GB | 448 GB/s | **128-bit** ❌ | ✅ + Tensor | $450+ | Q4_K_M 14B | **TRAP.** 38% slower than 2080 Ti. |
|
||
| RTX 4070 Ti Super | 16 GB | 672 GB/s | 256-bit | ✅ + Tensor | $600+ | Q4_K_M 20B | First good 16GB NVIDIA card. |
|
||
| RTX 3090 | 24 GB | 936 GB/s | 384-bit | ✅ + Tensor | $600-700 | Q4_K_M 32B | Real upgrade. Runs 32B comfortably. |
|
||
| RX 6650 XT | 8 GB | — | — | ❌ Vulkan only | ~$150 | — | Avoid for LLMs. |
|
||
| RX 7700 XT | 12 GB | 432 GB/s | — | ❌ Vulkan only | ~$350 | — | Avoid for LLMs. 2× 8-pin, 245W. |
|
||
|
||
## Key Specs That Matter for llama.cpp
|
||
|
||
- **VRAM**: The model must fit. For reference:
|
||
- Q4_K_M 7B ≈ 4.7 GB, Q5_K_M ≈ 5.5 GB, Q8_0 ≈ 8 GB
|
||
- Q4_K_M 14B ≈ 9 GB, Q3_K_M ≈ 6.5 GB, IQ3_M ≈ 7.5 GB
|
||
- Q3_K_M 20B ≈ 9 GB, IQ2_S ≈ 6 GB
|
||
- Q4_K_M 32B ≈ 18 GB, IQ2_S ≈ 8.5 GB
|
||
- Add ~1-2 GB for KV cache (context).
|
||
- **Two models at once**: Q4_K_M 7B (4.5 GB) + Q3_K_M 14B (6.5 GB) — barely fits 11GB.
|
||
- **Memory bandwidth**: #1 bottleneck. Every token reads the entire model from VRAM. **Bandwidth directly determines tokens/sec.** All else equal, a card with 50% more bandwidth generates tokens 50% faster for the same model.
|
||
- **Bus width**: A proxy for bandwidth. 128-bit cards universally bottleneck LLM workloads regardless of VRAM. 256-bit is the minimum for decent inference. 352-bit+ is where things get good.
|
||
- **CUDA compute capability**: 6.1+ needed for CUDA backend. 7.0+ (Volta and newer) gets full optimization. 7.5+ (Turing) gets tensor core acceleration.
|
||
- **Tensor cores**: Accelerate FP16 matrix ops. Measurable but modest speedup for llama.cpp CUDA backend.
|
||
|
||
## Power Cost Analysis
|
||
|
||
At typical US residential rates (~$0.13/kWh), GPU electricity cost is negligible. The idle draw difference between cards (15-30W) is a few cents per day. Even heavy inference (8 hours/day at full TDP) costs $3-13/month total for the entire card. The upfront purchase price dominates — not the power bill.
|
||
|
||
Rule of thumb: $80 GPU costs ~$2-3/month to run idle. $600 GPU costs ~$3-5/month idle. The difference is pocket change.
|
||
|
||
## nvidia-smi Power Limiting (Reduce Noise/Heat)
|
||
|
||
LLM inference is **memory-bandwidth-bound**, not core-clock-bound. You can throttle power with near-zero token speed loss:
|
||
|
||
```bash
|
||
# Check current power limit
|
||
nvidia-smi -q -d POWER | grep "Power Limit"
|
||
|
||
# Cap at 80% of max (keeps 95%+ token speed)
|
||
sudo nvidia-smi -pl 280 # 3090: 350W → 280W
|
||
sudo nvidia-smi -pl 200 # 2080 Ti: 250W → 200W
|
||
|
||
# Make persistent across reboots
|
||
sudo nvidia-smi -pm 1 # enable persistence mode
|
||
sudo nvidia-smi -pl 280 # set power limit
|
||
```
|
||
|
||
Blower cards (Turbo models) benefit most — lowering power from 350W to 280W drops fan RPM significantly because the small impeller screams above 60% speed. Triple-fan open-air cards benefit less since they're already quieter.
|
||
|
||
**Impact**: ~20% less power draw for ~3-5% slower tokens. On a 3090 running llama-server 24/7 in a living space, this is the difference between annoying and inaudible.
|
||
|
||
## Case Compatibility: Blower vs Open-Air
|
||
|
||
For prebuilt/server cases with limited airflow (HP Omen, Dell XPS, SFF builds), **blower-style cards are often better than open-air** despite being louder at stock power:
|
||
|
||
| Design | Heat exhaust | Best for | Noise |
|
||
|---|---|---|---|
|
||
| **Blower** (Turbo/Founders) | Out the back | Small cases, servers, prebuilts | Louder at full power |
|
||
| **Open-air** (FTW3/Gaming OC) | Into the case | Full towers, gaming cases | Quieter with good airflow |
|
||
|
||
**Why this matters**: An open-air 3090 dumps 350W into a cramped OMEN 30L — the CPU, VRMs, and drives all cook. The blower 3090 Turbo (267mm, 2-slot, 2×8-pin) fits the same case, exhausts heat out the rear, and runs quieter at a 280W power limit than at stock. The blower is also shorter (267mm vs 300mm FTW3) and needs fewer power cables (2×8-pin vs 3×8-pin).
|
||
|
||
**Before buying**: always check the case GPU length limit (`dmidecode -t chassis` or physical measurement) plus PSU connector count. Prebuilt PSUs often have fewer PCIe cables than aftermarket units.
|
||
|
||
## Server Power Measurement
|
||
|
||
For exact costs, a Kill-A-Watt meter (~$15) plugged between the wall and server gives real numbers. Software estimates vary — a 3090 system idling at "100W" from `nvidia-smi` might draw 130W at the wall after PSU inefficiency.
|
||
|
||
## System Profiling Commands
|
||
|
||
```bash
|
||
# GPU details
|
||
nvidia-smi --query-gpu=index,name,memory.total,power.limit --format=csv
|
||
sudo dmidecode -t baseboard | grep -E "Manufacturer:|Product Name:" # motherboard
|
||
sudo dmidecode -t system | grep -E "Manufacturer:|Product Name:" # system model
|
||
sudo dmidecode --type memory | grep -E "Type:|Speed:|Size:" # RAM
|
||
lscpu | grep "Model name" # CPU
|
||
|
||
# Check what's using VRAM
|
||
nvidia-smi
|
||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
||
```
|
||
|
||
## Recommendation Heuristic
|
||
|
||
1. **$80-100**: GTX 1080 8GB (320 GB/s) — best budget LLM card, CUDA native, faster than any 128-bit modern card
|
||
2. **$140-170**: GTX 1080 Ti 11GB (484 GB/s) — sweet spot, runs Q4_K_M 14B
|
||
3. **$250-300**: RTX 2080 Ti 11GB (616 GB/s) — best sub-$300 pick, tensor cores, near-3090 speed per dollar
|
||
4. **$350-400**: RTX 3080 12GB (912 GB/s) — major speed jump, 384-bit bus
|
||
5. **$600-700**: RTX 3090 24GB (936 GB/s) — runs 32B models, "buy once done"
|
||
6. **Skip entirely**: Any 128-bit NVIDIA card (4060 Ti, 5060 Ti) regardless of VRAM. You're paying for VRAM you can't feed fast enough.
|
||
7. **Skip entirely**: AMD consumer cards for LLM-first setups. Vulkan-only is not worth the tradeoffs.
|
||
8. **If you also game**: NVIDIA still wins (CUDA + DLSS for gaming, CUDA for LLMs).
|