initial commit

skills/mlops/inference/llama-cpp/references/vulkan-gpu-backend.md

@@ -0,0 +1,140 @@
+# Vulkan GPU Backend for llama.cpp
+
+Build and deploy llama.cpp with GPU acceleration via Vulkan — no CUDA toolkit required. Works on NVIDIA, AMD, and Intel GPUs with Vulkan drivers.
+
+## When to use Vulkan
+
+- **CUDA toolkit version mismatch** (nvcc 12.4 vs CUDA 13.1 libs) — Vulkan sidesteps it entirely
+- No CUDA toolkit installed and you don't want to install one
+- AMD or Intel GPU (ROCm not set up)
+- Want a single backend that works across GPU vendors
+
+## Performance
+
+Vulkan delivers 80-90% of CUDA inference speed for llama.cpp. For a 7B Q2_K model on a GTX 1050 Ti (4GB), expect ~11-12s per request at 4096 context.
+
+## Prerequisites
+
+```bash
+# Ubuntu/Debian
+sudo apt-get install -y libvulkan-dev glslc glslang-dev glslang-tools libglm-dev
+
+# glslc is the GLSL→SPIR-V shader compiler (critical — cmake fails with "Could NOT find Vulkan (missing: glslc)" without it)
+# It's available as the standalone 'glslc' package on Ubuntu 24.04+, or bundled in 'libshaderc-dev'
+# If 'glslc' package not found, use: sudo apt-get install -y libshaderc-dev
+# libglm-dev provides GLM math headers needed by the Vulkan shader compilation step
+
+# Check which package provides glslc on your distro:
+# apt-cache search glslc        # should show both 'glslc' and 'libshaderc-dev'
+# dpkg -S $(which glslc)        # find installed package
+```
+
+Also note: switching backends (CPU-only ↔ Vulkan) requires a fresh cmake configure. The cached build uses the previous backend:
+
+```bash
+# If you built CPU-only first, then want Vulkan:
+cd llama.cpp
+cmake -B build -DGGML_VULKAN=ON  # reconfigures from scratch
+cmake --build build -j8
+
+# If the cmake cache had GGML_VULKAN=OFF from a previous build,
+# you must explicitly set it ON — cmake remembers the old value
+```
+
+Verify Vulkan driver is loaded:
+```bash
+lsmod | grep nvidia   # or amdgpu for AMD
+```
+
+## Build from source
+
+```bash
+git clone https://github.com/ggml-org/llama.cpp
+cd llama.cpp
+cmake -B build -DGGML_VULKAN=ON
+cmake --build build -j8
+```
+
+Verify Vulkan was detected:
+```bash
+./build/bin/llama-server --help 2>&1 | grep -i vulkan
+# Should show: "ggml_vulkan: Found 1 Vulkan devices:"
+# Should list your GPU model
+```
+
+## Launch with full GPU offload
+
+```bash
+./build/bin/llama-server \
+  -m /path/to/model.gguf \
+  -c 4096 \
+  --host 127.0.0.1 \
+  --port 8081 \
+  -ngl 99 \
+  -t 8
+```
+
+- `-ngl 99`: offload all layers to GPU (use 99 to mean "everything")
+- `-t 8`: CPU threads for any remaining CPU work (KV cache management, tokenization)
+- If model doesn't fully fit in VRAM, reduce `-ngl` (e.g., `-ngl 22` for ~80% layers on GPU)
+
+## Systemd service
+
+```ini
+[Unit]
+Description=llama.cpp server with Vulkan GPU
+After=network.target
+
+[Service]
+Type=simple
+User=ray
+WorkingDirectory=/home/ray
+ExecStart=/home/ray/llama.cpp-build/build/bin/llama-server \
+    -m /home/ray/models/Qwen2.5-7B-Instruct-Q2_K.gguf \
+    -c 4096 \
+    --host 127.0.0.1 \
+    --port 8081 \
+    -ngl 99 \
+    -t 8
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target
+```
+
+## VRAM sizing
+
+How to estimate if a model fits in VRAM:
+
+| Model | Quant | File size | VRAM (~) | Fits 4GB? |
+|---|---|---|---|---|
+| Qwen2.5-7B | Q2_K | 3.0 GB | 3.2 GB | ✅ |
+| Qwen2.5-7B | Q3_K_M | 3.5 GB | 3.7 GB | ✅ |
+| Qwen2.5-7B | Q4_K_M | 4.7 GB | 5.0 GB | ❌ (partial offload) |
+| Qwen2.5-3B | Q4_K_M | 1.9 GB | 2.1 GB | ✅ |
+| Qwen2.5-3B | Q8_0 | 3.3 GB | 3.5 GB | ✅ |
+
+Rule of thumb: VRAM ≈ file size + 200-300MB for KV cache at 4096 context.
+
+## Detection and verification
+
+```bash
+# List Vulkan devices detected by llama.cpp
+./build/bin/llama-server --help 2>&1 | grep "ggml_vulkan"
+
+# Example output:
+# ggml_vulkan: Found 1 Vulkan devices:
+# ggml_vulkan: 0 = NVIDIA GeForce GTX 1050 Ti (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32
+```
+
+Check GPU memory usage during inference:
+```bash
+nvidia-smi  # NVIDIA
+# or
+radeontop   # AMD
+```
+
+## Switching back to CPU-only
+
+Remove `-ngl 99` from the command. The same Vulkan-built binary works for CPU — it just won't offload any layers.