ray
141 lines
3.9 KiB
Markdown
141 lines
3.9 KiB
Markdown
# Vulkan GPU Backend for llama.cpp
|
|
|
|
Build and deploy llama.cpp with GPU acceleration via Vulkan — no CUDA toolkit required. Works on NVIDIA, AMD, and Intel GPUs with Vulkan drivers.
|
|
|
|
## When to use Vulkan
|
|
|
|
- **CUDA toolkit version mismatch** (nvcc 12.4 vs CUDA 13.1 libs) — Vulkan sidesteps it entirely
|
|
- No CUDA toolkit installed and you don't want to install one
|
|
- AMD or Intel GPU (ROCm not set up)
|
|
- Want a single backend that works across GPU vendors
|
|
|
|
## Performance
|
|
|
|
Vulkan delivers 80-90% of CUDA inference speed for llama.cpp. For a 7B Q2_K model on a GTX 1050 Ti (4GB), expect ~11-12s per request at 4096 context.
|
|
|
|
## Prerequisites
|
|
|
|
```bash
|
|
# Ubuntu/Debian
|
|
sudo apt-get install -y libvulkan-dev glslc glslang-dev glslang-tools libglm-dev
|
|
|
|
# glslc is the GLSL→SPIR-V shader compiler (critical — cmake fails with "Could NOT find Vulkan (missing: glslc)" without it)
|
|
# It's available as the standalone 'glslc' package on Ubuntu 24.04+, or bundled in 'libshaderc-dev'
|
|
# If 'glslc' package not found, use: sudo apt-get install -y libshaderc-dev
|
|
# libglm-dev provides GLM math headers needed by the Vulkan shader compilation step
|
|
|
|
# Check which package provides glslc on your distro:
|
|
# apt-cache search glslc # should show both 'glslc' and 'libshaderc-dev'
|
|
# dpkg -S $(which glslc) # find installed package
|
|
```
|
|
|
|
Also note: switching backends (CPU-only ↔ Vulkan) requires a fresh cmake configure. The cached build uses the previous backend:
|
|
|
|
```bash
|
|
# If you built CPU-only first, then want Vulkan:
|
|
cd llama.cpp
|
|
cmake -B build -DGGML_VULKAN=ON # reconfigures from scratch
|
|
cmake --build build -j8
|
|
|
|
# If the cmake cache had GGML_VULKAN=OFF from a previous build,
|
|
# you must explicitly set it ON — cmake remembers the old value
|
|
```
|
|
|
|
Verify Vulkan driver is loaded:
|
|
```bash
|
|
lsmod | grep nvidia # or amdgpu for AMD
|
|
```
|
|
|
|
## Build from source
|
|
|
|
```bash
|
|
git clone https://github.com/ggml-org/llama.cpp
|
|
cd llama.cpp
|
|
cmake -B build -DGGML_VULKAN=ON
|
|
cmake --build build -j8
|
|
```
|
|
|
|
Verify Vulkan was detected:
|
|
```bash
|
|
./build/bin/llama-server --help 2>&1 | grep -i vulkan
|
|
# Should show: "ggml_vulkan: Found 1 Vulkan devices:"
|
|
# Should list your GPU model
|
|
```
|
|
|
|
## Launch with full GPU offload
|
|
|
|
```bash
|
|
./build/bin/llama-server \
|
|
-m /path/to/model.gguf \
|
|
-c 4096 \
|
|
--host 127.0.0.1 \
|
|
--port 8081 \
|
|
-ngl 99 \
|
|
-t 8
|
|
```
|
|
|
|
- `-ngl 99`: offload all layers to GPU (use 99 to mean "everything")
|
|
- `-t 8`: CPU threads for any remaining CPU work (KV cache management, tokenization)
|
|
- If model doesn't fully fit in VRAM, reduce `-ngl` (e.g., `-ngl 22` for ~80% layers on GPU)
|
|
|
|
## Systemd service
|
|
|
|
```ini
|
|
[Unit]
|
|
Description=llama.cpp server with Vulkan GPU
|
|
After=network.target
|
|
|
|
[Service]
|
|
Type=simple
|
|
User=ray
|
|
WorkingDirectory=/home/ray
|
|
ExecStart=/home/ray/llama.cpp-build/build/bin/llama-server \
|
|
-m /home/ray/models/Qwen2.5-7B-Instruct-Q2_K.gguf \
|
|
-c 4096 \
|
|
--host 127.0.0.1 \
|
|
--port 8081 \
|
|
-ngl 99 \
|
|
-t 8
|
|
Restart=on-failure
|
|
RestartSec=5
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
```
|
|
|
|
## VRAM sizing
|
|
|
|
How to estimate if a model fits in VRAM:
|
|
|
|
| Model | Quant | File size | VRAM (~) | Fits 4GB? |
|
|
|---|---|---|---|---|
|
|
| Qwen2.5-7B | Q2_K | 3.0 GB | 3.2 GB | ✅ |
|
|
| Qwen2.5-7B | Q3_K_M | 3.5 GB | 3.7 GB | ✅ |
|
|
| Qwen2.5-7B | Q4_K_M | 4.7 GB | 5.0 GB | ❌ (partial offload) |
|
|
| Qwen2.5-3B | Q4_K_M | 1.9 GB | 2.1 GB | ✅ |
|
|
| Qwen2.5-3B | Q8_0 | 3.3 GB | 3.5 GB | ✅ |
|
|
|
|
Rule of thumb: VRAM ≈ file size + 200-300MB for KV cache at 4096 context.
|
|
|
|
## Detection and verification
|
|
|
|
```bash
|
|
# List Vulkan devices detected by llama.cpp
|
|
./build/bin/llama-server --help 2>&1 | grep "ggml_vulkan"
|
|
|
|
# Example output:
|
|
# ggml_vulkan: Found 1 Vulkan devices:
|
|
# ggml_vulkan: 0 = NVIDIA GeForce GTX 1050 Ti (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32
|
|
```
|
|
|
|
Check GPU memory usage during inference:
|
|
```bash
|
|
nvidia-smi # NVIDIA
|
|
# or
|
|
radeontop # AMD
|
|
```
|
|
|
|
## Switching back to CPU-only
|
|
|
|
Remove `-ngl 99` from the command. The same Vulkan-built binary works for CPU — it just won't offload any layers.
|