hermes-config/skills/self-hosting/docker-gpu-acceleration/SKILL.md

name, description, category

name	description	category
docker-gpu-acceleration	Set up, verify, and troubleshoot NVIDIA GPU acceleration for Docker containers running ML/AI services (Immich ML, ONNX models, LLMs, etc.)	self-hosting

Docker GPU Acceleration

Set up NVIDIA GPU access in Docker containers for ML/AI workloads and debug when it isn't working.

When to use

• User wants GPU acceleration in Docker for Immich ML, LLM serving, or ONNX inference
• nvidia-smi shows 0% util / 11 MiB / no processes — GPU idle when it should be working
• Container repeatedly fails to load ML models (download→fail→clear→retry loop)
• Image uses -cuda suffix but GPU isn't actually being used

Steps

1. Verify host GPU is functional

nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv,noheader

Idle baseline: ~11 MiB memory, 0% util, P8 power state Active: >100 MiB memory, >0% util, P0 power state

2. Check Docker nvidia runtime is available

docker info | grep -i "runtimes"

Must show nvidia in the list. If not, install nvidia-container-toolkit:

apt install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

3. Add GPU access to docker-compose.yml

Add to the service that needs the GPU:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

Then recreate: docker compose up -d <service>

4. Verify GPU access inside container

# Check NVIDIA devices
docker exec <container> ls -la /dev | grep nvidia

# Should show: nvidia0, nvidiactl, nvidia-uvm, nvidia-caps (driver 580+)

# Check caps specifically (driver 580+ requirement)
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
# If "No such file or directory" → CDI spec is missing caps. See references/cdi-caps-fix.md

# Check ONNX Runtime sees GPU
docker exec <container> python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"

# Should show: GPU and ['CUDAExecutionProvider', 'TensorrtExecutionProvider', 'CPUExecutionProvider']

5. Test actual GPU inference

Create a minimal ONNX model and run it with CUDAExecutionProvider to verify the GPU executes work, not just reports as available.

6. Pre-cache models to avoid download loops

If the service downloads→fails→clears→retries in a loop, the model cache is empty. Download manually:

from huggingface_hub import snapshot_download
result = snapshot_download(
    "immich-app/<model_name>",
    cache_dir="/cache/<model_task>/<model_name>",
    local_dir="/cache/<model_task>/<model_name>",
    ignore_patterns=["*.armnn", "*.rknn"],
)

Then restart the container — it picks up cached models and loads immediately.

7. Monitor GPU utilization

nvidia-smi  # snapshot view
nvtop       # live TUI (install with apt install nvtop)

8. (Optional) Cockpit Web UI Dashboard

Create a custom Cockpit package that shows live GPU metrics in the web UI sidebar at https://<server>:9090:

sudo mkdir -p /usr/share/cockpit/nvidia-gpu

⚠️ Important: cockpit.script() (run nvidia-smi directly) silently fails on Cockpit v314+ (Ubuntu 26+). The recommended approach is a systemd service that writes GPU data to /run/*.txt files, then the Cockpit page reads them via cockpit.file().read().

Follow the full instructions in references/cockpit-gpu-dashboard.md — it documents both approaches with the correct working procedure.

sudo systemctl restart cockpit

The page displays model, driver, CUDA version, temperature, utilization, memory bar, power draw, and running processes — refreshing every 5 seconds.

9. Verify GPU Dashboard

After setup, run the verification script:

~/.hermes/skills/self-hosting/docker-gpu-acceleration/scripts/verify-cockpit-gpu.sh

This checks the systemd service, data files, Cockpit package installation, nvidia-smi accessibility, and Cockpit service health.

Pitfalls

• Compose shows Runtime: runc and DeviceRequests: null — GPU was never configured. Add deploy.resources.reservations.devices to the service.
• ONNX Runtime reports "no CUDA-capable device is detected" / nvidia-smi fails inside container — Likely missing /dev/nvidia-caps/. Check docker exec <container> ls /dev/nvidia-caps/. If absent despite host having them, the CDI spec (at /var/run/cdi/nvidia.yaml) is missing caps device entries. See references/cdi-caps-fix.md for the fix. Driver 580+ requires caps for CUDA initialization.
• Model download→fail→clear→retry loop — Cache directory is empty. Pre-download models, then restart container.
• EHOSTUNREACH between containers on same bridge — Docker bridge networking glitch. docker compose restart database redis server to fix.
• Model download returns 401 — Don't use raw HuggingFace URL. Use huggingface_hub.snapshot_download() Python API, which handles auth correctly.
• Container CUDA 12.2 on host CUDA 13.0 — Usually fine (CUDA backward-compatible within major versions), but if models fail to load, check the exact error.
• 4 GB VRAM limit — Some GPU-inference stacks need all models loaded simultaneously. If VRAM fills, consider MACHINE_LEARNING_MODEL_ARENA=true (loads one model at a time).
• nvtop not available in apt — Build from source at https://github.com/Syllo/nvtop.

Verification checklist

• nvidia-smi shows python process using GPU memory
• Service container logs show successful model loading (no "Failed to load" warnings)
• Service health check passes
• Job queues show active / waiting items (not stuck at 0)
• GPU temp rises above idle (28-30°C → 40-65°C under load)

Related

• references/immich-ml-gpu-paths.md — Exact cache paths and model names for Immich ML
• references/cockpit-gpu-dashboard.md — Cockpit web UI GPU dashboard package
• references/cdi-caps-fix.md — CDI spec missing /dev/nvidia-caps/ on driver 580+ (CUDA fails, ONNX falls back to CPU)