GPU Hardware Recommendations for llama.cpp

Hardware analysis for selecting a GPU for local llama.cpp inference. All analysis assumes llama.cpp as the inference engine.

The Golden Rule: CUDA is Everything

For llama.cpp inference, NVIDIA CUDA is the only sane choice for consumer GPUs. AMD consumer cards (RDNA2/RDNA3) are locked to Vulkan backend — no ROCm support on consumer SKUs. The Vulkan backend works but has real limitations:

No flash attention
No MMQ kernels (slower quants)
More bugs, less optimization priority from the llama.cpp team
Real-world token generation is often on par with or slower than a weaker NVIDIA card on CUDA, despite higher raw bandwidth

AMD integrated GPUs and iGPUs: Vulkan works fine for these (they're small anyway). The problem is discrete AMD consumer cards where you're paying for bandwidth you can't fully use.

The 128-Bit Bus Trap

NVIDIA's xx60 series (4060, 4060 Ti, 5060, 5060 Ti) uses a crippled 128-bit memory bus — half the width of older cards like the GTX 1080 (256-bit) or RTX 2080 Ti (352-bit). Even with GDDR6X/GDDR7 speeds, the narrow bus caps effective bandwidth so severely that a $400 RTX 4060 Ti 16GB (288 GB/s) is slower than an $80 GTX 1080 (320 GB/s) for LLM token generation.

The 16GB dead zone: the first NVIDIA 16GB card with a bus wide enough for LLMs is the RTX 4070 Ti Super (256-bit, 672 GB/s) at ~$600. Every 16GB card below it is a 128-bit gaming card masquerading as an AI card — attractive VRAM number, useless bandwidth.

Comparison Table

GPU	VRAM	Bandwidth	Bus	CUDA	Price (used)	Best model fits	Notes
GTX 1050 Ti	4 GB	112 GB/s	128-bit	❌ Vulkan only	—	Q2_K 7B	Slot-powered. Baseline.
GTX 1070	8 GB	256 GB/s	256-bit	✅	~$80-100	Q4_K_M 7B, Q2 14B	1× 8-pin, 150W.
GTX 1080	8 GB	320 GB/s	256-bit	✅	~$80	Q5_K_M 7B	Best budget LLM card. Faster than 4060 Ti 16GB.
GTX 1080 Ti	11 GB	484 GB/s	352-bit	✅	~$140-170	Q4_K_M 14B	Sweet spot. 11GB + 484 GB/s.
RTX 2080 Ti	11 GB	616 GB/s	352-bit	✅ + Tensor	~$250-300	Q4_K_M 14B, IQ3 20B	Best sub-$300 card. 2× RTX 2060 12GB speed.
RTX 2060 12GB	12 GB	336 GB/s	192-bit	✅ + Tensor	~$200	Q4_K_M 14B	1GB more than 1080 Ti but 44% slower. VLMs need 12GB+.
RTX 2060 SUPER	8 GB	448 GB/s	256-bit	✅ + Tensor	~$180-220	Q5_K_M 7B	Fast 7B card, hard-capped at 8GB.
RTX 3070 Ti	8 GB	608 GB/s	256-bit	✅ + Tensor	~$250-300	Q5_K_M 7B	Very fast, but 8GB only — same ceiling as $80 GTX 1080.
RTX 3080 10GB	10 GB	760 GB/s	320-bit	✅ + Tensor	~$350-400	Q4_K_M 14B	Faster than 2080 Ti but 1GB less VRAM.
RTX 3080 12GB	12 GB	912 GB/s	384-bit	✅ + Tensor	~$350-400	Q4_K_M 14B+	True upgrade from 2080 Ti. 48% faster.
RTX 4060 Ti 16GB	16 GB	288 GB/s	128-bit ❌	✅ + Tensor	$400+	Q4_K_M 14B	TRAP. Slower than $80 GTX 1080.
RTX 5060 Ti 16GB	16 GB	448 GB/s	128-bit ❌	✅ + Tensor	$450+	Q4_K_M 14B	TRAP. 38% slower than 2080 Ti.
RTX 4070 Ti Super	16 GB	672 GB/s	256-bit	✅ + Tensor	$600+	Q4_K_M 20B	First good 16GB NVIDIA card.
RTX 3090	24 GB	936 GB/s	384-bit	✅ + Tensor	$600-700	Q4_K_M 32B	Real upgrade. Runs 32B comfortably.
RX 6650 XT	8 GB	—	—	❌ Vulkan only	~$150	—	Avoid for LLMs.
RX 7700 XT	12 GB	432 GB/s	—	❌ Vulkan only	~$350	—	Avoid for LLMs. 2× 8-pin, 245W.

Key Specs That Matter for llama.cpp

VRAM: The model must fit. For reference:
- Q4_K_M 7B ≈ 4.7 GB, Q5_K_M ≈ 5.5 GB, Q8_0 ≈ 8 GB
- Q4_K_M 14B ≈ 9 GB, Q3_K_M ≈ 6.5 GB, IQ3_M ≈ 7.5 GB
- Q3_K_M 20B ≈ 9 GB, IQ2_S ≈ 6 GB
- Q4_K_M 32B ≈ 18 GB, IQ2_S ≈ 8.5 GB
- Add ~1-2 GB for KV cache (context).
- Two models at once: Q4_K_M 7B (4.5 GB) + Q3_K_M 14B (6.5 GB) — barely fits 11GB.
Memory bandwidth: #1 bottleneck. Every token reads the entire model from VRAM. Bandwidth directly determines tokens/sec. All else equal, a card with 50% more bandwidth generates tokens 50% faster for the same model.
Bus width: A proxy for bandwidth. 128-bit cards universally bottleneck LLM workloads regardless of VRAM. 256-bit is the minimum for decent inference. 352-bit+ is where things get good.
CUDA compute capability: 6.1+ needed for CUDA backend. 7.0+ (Volta and newer) gets full optimization. 7.5+ (Turing) gets tensor core acceleration.
Tensor cores: Accelerate FP16 matrix ops. Measurable but modest speedup for llama.cpp CUDA backend.

Power Cost Analysis

At typical US residential rates (~$0.13/kWh), GPU electricity cost is negligible. The idle draw difference between cards (15-30W) is a few cents per day. Even heavy inference (8 hours/day at full TDP) costs $3-13/month total for the entire card. The upfront purchase price dominates — not the power bill.

Rule of thumb: $80 GPU costs ~$2-3/month to run idle. $600 GPU costs ~$3-5/month idle. The difference is pocket change.

nvidia-smi Power Limiting (Reduce Noise/Heat)

LLM inference is memory-bandwidth-bound, not core-clock-bound. You can throttle power with near-zero token speed loss:

# Check current power limit
nvidia-smi -q -d POWER | grep "Power Limit"

# Cap at 80% of max (keeps 95%+ token speed)
sudo nvidia-smi -pl 280    # 3090: 350W → 280W
sudo nvidia-smi -pl 200    # 2080 Ti: 250W → 200W

# Make persistent across reboots
sudo nvidia-smi -pm 1       # enable persistence mode
sudo nvidia-smi -pl 280     # set power limit

Blower cards (Turbo models) benefit most — lowering power from 350W to 280W drops fan RPM significantly because the small impeller screams above 60% speed. Triple-fan open-air cards benefit less since they're already quieter.

Impact: ~20% less power draw for ~3-5% slower tokens. On a 3090 running llama-server 24/7 in a living space, this is the difference between annoying and inaudible.

Case Compatibility: Blower vs Open-Air

For prebuilt/server cases with limited airflow (HP Omen, Dell XPS, SFF builds), blower-style cards are often better than open-air despite being louder at stock power:

Design	Heat exhaust	Best for	Noise
Blower (Turbo/Founders)	Out the back	Small cases, servers, prebuilts	Louder at full power
Open-air (FTW3/Gaming OC)	Into the case	Full towers, gaming cases	Quieter with good airflow

Why this matters: An open-air 3090 dumps 350W into a cramped OMEN 30L — the CPU, VRMs, and drives all cook. The blower 3090 Turbo (267mm, 2-slot, 2×8-pin) fits the same case, exhausts heat out the rear, and runs quieter at a 280W power limit than at stock. The blower is also shorter (267mm vs 300mm FTW3) and needs fewer power cables (2×8-pin vs 3×8-pin).

Before buying: always check the case GPU length limit (dmidecode -t chassis or physical measurement) plus PSU connector count. Prebuilt PSUs often have fewer PCIe cables than aftermarket units.

Server Power Measurement

For exact costs, a Kill-A-Watt meter (~$15) plugged between the wall and server gives real numbers. Software estimates vary — a 3090 system idling at "100W" from nvidia-smi might draw 130W at the wall after PSU inefficiency.

System Profiling Commands

# GPU details
nvidia-smi --query-gpu=index,name,memory.total,power.limit --format=csv
sudo dmidecode -t baseboard | grep -E "Manufacturer:|Product Name:"  # motherboard
sudo dmidecode -t system | grep -E "Manufacturer:|Product Name:"  # system model
sudo dmidecode --type memory | grep -E "Type:|Speed:|Size:"  # RAM
lscpu | grep "Model name"  # CPU

# Check what's using VRAM
nvidia-smi
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv

Recommendation Heuristic

$80-100: GTX 1080 8GB (320 GB/s) — best budget LLM card, CUDA native, faster than any 128-bit modern card
$140-170: GTX 1080 Ti 11GB (484 GB/s) — sweet spot, runs Q4_K_M 14B
$250-300: RTX 2080 Ti 11GB (616 GB/s) — best sub-$300 pick, tensor cores, near-3090 speed per dollar
$350-400: RTX 3080 12GB (912 GB/s) — major speed jump, 384-bit bus
$600-700: RTX 3090 24GB (936 GB/s) — runs 32B models, "buy once done"
Skip entirely: Any 128-bit NVIDIA card (4060 Ti, 5060 Ti) regardless of VRAM. You're paying for VRAM you can't feed fast enough.
Skip entirely: AMD consumer cards for LLM-first setups. Vulkan-only is not worth the tradeoffs.
If you also game: NVIDIA still wins (CUDA + DLSS for gaming, CUDA for LLMs).

8.5 KiB Raw Permalink Blame History Unescape Escape