---
name: docker-gpu-acceleration
description: Set up, verify, and troubleshoot NVIDIA GPU acceleration for Docker containers running ML/AI services (Immich ML, ONNX models, LLMs, etc.)
category: self-hosting
---

# Docker GPU Acceleration

Set up NVIDIA GPU access in Docker containers for ML/AI workloads and debug when it isn't working.

## When to use

- User wants GPU acceleration in Docker for Immich ML, LLM serving, or ONNX inference
- `nvidia-smi` shows 0% util / 11 MiB / no processes — GPU idle when it should be working
- Container repeatedly fails to load ML models (download→fail→clear→retry loop)
- Image uses `-cuda` suffix but GPU isn't actually being used

## Steps

### 1. Verify host GPU is functional

```bash
nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv,noheader
```

**Idle baseline**: ~11 MiB memory, 0% util, P8 power state
**Active**: >100 MiB memory, >0% util, P0 power state

### 2. Check Docker nvidia runtime is available

```bash
docker info | grep -i "runtimes"
```

Must show `nvidia` in the list. If not, install `nvidia-container-toolkit`:

```bash
apt install nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
```

### 3. Add GPU access to docker-compose.yml

Add to the service that needs the GPU:

```yaml
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]
```

Then recreate: `docker compose up -d <service>`

### 4. Verify GPU access inside container

```bash
# Check NVIDIA devices
docker exec <container> ls -la /dev | grep nvidia

# Should show: nvidia0, nvidiactl, nvidia-uvm, nvidia-caps (driver 580+)

# Check caps specifically (driver 580+ requirement)
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
# If "No such file or directory" → CDI spec is missing caps. See references/cdi-caps-fix.md

# Check ONNX Runtime sees GPU
docker exec <container> python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"

# Should show: GPU and ['CUDAExecutionProvider', 'TensorrtExecutionProvider', 'CPUExecutionProvider']
```

### 5. Test actual GPU inference

Create a minimal ONNX model and run it with CUDAExecutionProvider to verify the GPU executes work, not just reports as available.

### 6. Pre-cache models to avoid download loops

If the service downloads→fails→clears→retries in a loop, the model cache is empty. Download manually:

```python
from huggingface_hub import snapshot_download
result = snapshot_download(
    "immich-app/<model_name>",
    cache_dir="/cache/<model_task>/<model_name>",
    local_dir="/cache/<model_task>/<model_name>",
    ignore_patterns=["*.armnn", "*.rknn"],
)
```

Then restart the container — it picks up cached models and loads immediately.

### 7. Monitor GPU utilization

```bash
nvidia-smi  # snapshot view
nvtop       # live TUI (install with apt install nvtop)
```

### 8. (Optional) Cockpit Web UI Dashboard

Create a custom Cockpit package that shows live GPU metrics in the web UI
sidebar at `https://<server>:9090`:

```bash
sudo mkdir -p /usr/share/cockpit/nvidia-gpu
```

**⚠️ Important:** `cockpit.script()` (run nvidia-smi directly) silently fails
on Cockpit v314+ (Ubuntu 26+). The **recommended approach** is a systemd
service that writes GPU data to `/run/*.txt` files, then the Cockpit page
reads them via `cockpit.file().read()`.

Follow the full instructions in `references/cockpit-gpu-dashboard.md` — it
documents both approaches with the correct working procedure.

```bash
sudo systemctl restart cockpit
```

The page displays model, driver, CUDA version, temperature, utilization,
memory bar, power draw, and running processes — refreshing every 5 seconds.

### 9. Verify GPU Dashboard

After setup, run the verification script:

```bash
~/.hermes/skills/self-hosting/docker-gpu-acceleration/scripts/verify-cockpit-gpu.sh
```

This checks the systemd service, data files, Cockpit package installation,
nvidia-smi accessibility, and Cockpit service health.

## Pitfalls

- **Compose shows `Runtime: runc` and `DeviceRequests: null`** — GPU was never configured. Add `deploy.resources.reservations.devices` to the service.
- **ONNX Runtime reports "no CUDA-capable device is detected" / nvidia-smi fails inside container** — Likely missing `/dev/nvidia-caps/`. Check `docker exec <container> ls /dev/nvidia-caps/`. If absent despite host having them, the CDI spec (at `/var/run/cdi/nvidia.yaml`) is missing caps device entries. See `references/cdi-caps-fix.md` for the fix. Driver 580+ requires caps for CUDA initialization.
- **Model download→fail→clear→retry loop** — Cache directory is empty. Pre-download models, then restart container.
- **EHOSTUNREACH between containers on same bridge** — Docker bridge networking glitch. `docker compose restart database redis server` to fix.
- **Model download returns 401** — Don't use raw HuggingFace URL. Use `huggingface_hub.snapshot_download()` Python API, which handles auth correctly.
- **Container CUDA 12.2 on host CUDA 13.0** — Usually fine (CUDA backward-compatible within major versions), but if models fail to load, check the exact error.
- **4 GB VRAM limit** — Some GPU-inference stacks need all models loaded simultaneously. If VRAM fills, consider `MACHINE_LEARNING_MODEL_ARENA=true` (loads one model at a time).
- **nvtop not available in apt** — Build from source at `https://github.com/Syllo/nvtop`.

## Verification checklist

- [ ] `nvidia-smi` shows python process using GPU memory
- [ ] Service container logs show successful model loading (no "Failed to load" warnings)
- [ ] Service health check passes
- [ ] Job queues show active / waiting items (not stuck at 0)
- [ ] GPU temp rises above idle (28-30°C → 40-65°C under load)

## Related

- `references/immich-ml-gpu-paths.md` — Exact cache paths and model names for Immich ML
- `references/cockpit-gpu-dashboard.md` — Cockpit web UI GPU dashboard package
- `references/cdi-caps-fix.md` — CDI spec missing `/dev/nvidia-caps/` on driver 580+ (CUDA fails, ONNX falls back to CPU)