Vulkan GPU Backend for llama.cpp

Build and deploy llama.cpp with GPU acceleration via Vulkan — no CUDA toolkit required. Works on NVIDIA, AMD, and Intel GPUs with Vulkan drivers.

When to use Vulkan

CUDA toolkit version mismatch (nvcc 12.4 vs CUDA 13.1 libs) — Vulkan sidesteps it entirely
No CUDA toolkit installed and you don't want to install one
AMD or Intel GPU (ROCm not set up)
Want a single backend that works across GPU vendors

Performance

Vulkan delivers 80-90% of CUDA inference speed for llama.cpp. For a 7B Q2_K model on a GTX 1050 Ti (4GB), expect ~11-12s per request at 4096 context.

Prerequisites

# Ubuntu/Debian
sudo apt-get install -y libvulkan-dev glslc glslang-dev glslang-tools libglm-dev

# glslc is the GLSL→SPIR-V shader compiler (critical — cmake fails with "Could NOT find Vulkan (missing: glslc)" without it)
# It's available as the standalone 'glslc' package on Ubuntu 24.04+, or bundled in 'libshaderc-dev'
# If 'glslc' package not found, use: sudo apt-get install -y libshaderc-dev
# libglm-dev provides GLM math headers needed by the Vulkan shader compilation step

# Check which package provides glslc on your distro:
# apt-cache search glslc        # should show both 'glslc' and 'libshaderc-dev'
# dpkg -S $(which glslc)        # find installed package

Also note: switching backends (CPU-only ↔ Vulkan) requires a fresh cmake configure. The cached build uses the previous backend:

# If you built CPU-only first, then want Vulkan:
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON  # reconfigures from scratch
cmake --build build -j8

# If the cmake cache had GGML_VULKAN=OFF from a previous build,
# you must explicitly set it ON — cmake remembers the old value

Verify Vulkan driver is loaded:

lsmod | grep nvidia   # or amdgpu for AMD

Build from source

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON
cmake --build build -j8

Verify Vulkan was detected:

./build/bin/llama-server --help 2>&1 | grep -i vulkan
# Should show: "ggml_vulkan: Found 1 Vulkan devices:"
# Should list your GPU model

Launch with full GPU offload

./build/bin/llama-server \
  -m /path/to/model.gguf \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8081 \
  -ngl 99 \
  -t 8

-ngl 99: offload all layers to GPU (use 99 to mean "everything")
-t 8: CPU threads for any remaining CPU work (KV cache management, tokenization)
If model doesn't fully fit in VRAM, reduce -ngl (e.g., -ngl 22 for ~80% layers on GPU)

Systemd service

[Unit]
Description=llama.cpp server with Vulkan GPU
After=network.target

[Service]
Type=simple
User=ray
WorkingDirectory=/home/ray
ExecStart=/home/ray/llama.cpp-build/build/bin/llama-server \
    -m /home/ray/models/Qwen2.5-7B-Instruct-Q2_K.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -ngl 99 \
    -t 8
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

VRAM sizing

How to estimate if a model fits in VRAM:

Model	Quant	File size	VRAM (~)	Fits 4GB?
Qwen2.5-7B	Q2_K	3.0 GB	3.2 GB	✅
Qwen2.5-7B	Q3_K_M	3.5 GB	3.7 GB	✅
Qwen2.5-7B	Q4_K_M	4.7 GB	5.0 GB	❌ (partial offload)
Qwen2.5-3B	Q4_K_M	1.9 GB	2.1 GB	✅
Qwen2.5-3B	Q8_0	3.3 GB	3.5 GB	✅

Rule of thumb: VRAM ≈ file size + 200-300MB for KV cache at 4096 context.

Detection and verification

# List Vulkan devices detected by llama.cpp
./build/bin/llama-server --help 2>&1 | grep "ggml_vulkan"

# Example output:
# ggml_vulkan: Found 1 Vulkan devices:
# ggml_vulkan: 0 = NVIDIA GeForce GTX 1050 Ti (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32

Check GPU memory usage during inference:

nvidia-smi  # NVIDIA
# or
radeontop   # AMD

Switching back to CPU-only

Remove -ngl 99 from the command. The same Vulkan-built binary works for CPU — it just won't offload any layers.

3.9 KiB Raw Blame History