# Vulkan GPU Backend for llama.cpp

Build and deploy llama.cpp with GPU acceleration via Vulkan — no CUDA toolkit required. Works on NVIDIA, AMD, and Intel GPUs with Vulkan drivers.

## When to use Vulkan

- **CUDA toolkit version mismatch** (nvcc 12.4 vs CUDA 13.1 libs) — Vulkan sidesteps it entirely
- No CUDA toolkit installed and you don't want to install one
- AMD or Intel GPU (ROCm not set up)
- Want a single backend that works across GPU vendors

## Performance

Vulkan delivers 80-90% of CUDA inference speed for llama.cpp. For a 7B Q2_K model on a GTX 1050 Ti (4GB), expect ~11-12s per request at 4096 context.

## Prerequisites

```bash
# Ubuntu/Debian
sudo apt-get install -y libvulkan-dev glslc glslang-dev glslang-tools libglm-dev

# glslc is the GLSL→SPIR-V shader compiler (critical — cmake fails with "Could NOT find Vulkan (missing: glslc)" without it)
# It's available as the standalone 'glslc' package on Ubuntu 24.04+, or bundled in 'libshaderc-dev'
# If 'glslc' package not found, use: sudo apt-get install -y libshaderc-dev
# libglm-dev provides GLM math headers needed by the Vulkan shader compilation step

# Check which package provides glslc on your distro:
# apt-cache search glslc        # should show both 'glslc' and 'libshaderc-dev'
# dpkg -S $(which glslc)        # find installed package
```

Also note: switching backends (CPU-only ↔ Vulkan) requires a fresh cmake configure. The cached build uses the previous backend:

```bash
# If you built CPU-only first, then want Vulkan:
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON  # reconfigures from scratch
cmake --build build -j8

# If the cmake cache had GGML_VULKAN=OFF from a previous build,
# you must explicitly set it ON — cmake remembers the old value
```

Verify Vulkan driver is loaded:
```bash
lsmod | grep nvidia   # or amdgpu for AMD
```

## Build from source

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_VULKAN=ON
cmake --build build -j8
```

Verify Vulkan was detected:
```bash
./build/bin/llama-server --help 2>&1 | grep -i vulkan
# Should show: "ggml_vulkan: Found 1 Vulkan devices:"
# Should list your GPU model
```

## Launch with full GPU offload

```bash
./build/bin/llama-server \
  -m /path/to/model.gguf \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8081 \
  -ngl 99 \
  -t 8
```

- `-ngl 99`: offload all layers to GPU (use 99 to mean "everything")
- `-t 8`: CPU threads for any remaining CPU work (KV cache management, tokenization)
- If model doesn't fully fit in VRAM, reduce `-ngl` (e.g., `-ngl 22` for ~80% layers on GPU)

## Systemd service

```ini
[Unit]
Description=llama.cpp server with Vulkan GPU
After=network.target

[Service]
Type=simple
User=ray
WorkingDirectory=/home/ray
ExecStart=/home/ray/llama.cpp-build/build/bin/llama-server \
    -m /home/ray/models/Qwen2.5-7B-Instruct-Q2_K.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -ngl 99 \
    -t 8
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

## VRAM sizing

How to estimate if a model fits in VRAM:

| Model | Quant | File size | VRAM (~) | Fits 4GB? |
|---|---|---|---|---|
| Qwen2.5-7B | Q2_K | 3.0 GB | 3.2 GB | ✅ |
| Qwen2.5-7B | Q3_K_M | 3.5 GB | 3.7 GB | ✅ |
| Qwen2.5-7B | Q4_K_M | 4.7 GB | 5.0 GB | ❌ (partial offload) |
| Qwen2.5-3B | Q4_K_M | 1.9 GB | 2.1 GB | ✅ |
| Qwen2.5-3B | Q8_0 | 3.3 GB | 3.5 GB | ✅ |

Rule of thumb: VRAM ≈ file size + 200-300MB for KV cache at 4096 context.

## Detection and verification

```bash
# List Vulkan devices detected by llama.cpp
./build/bin/llama-server --help 2>&1 | grep "ggml_vulkan"

# Example output:
# ggml_vulkan: Found 1 Vulkan devices:
# ggml_vulkan: 0 = NVIDIA GeForce GTX 1050 Ti (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32
```

Check GPU memory usage during inference:
```bash
nvidia-smi  # NVIDIA
# or
radeontop   # AMD
```

## Switching back to CPU-only

Remove `-ngl 99` from the command. The same Vulkan-built binary works for CPU — it just won't offload any layers.