hermes-config/skills/self-hosting/docker-gpu-acceleration/references/cdi-caps-fix.md

CDI Caps Device Fix (Driver 580+)

The Problem

NVIDIA driver 580+ requires /dev/nvidia-caps/nvidia-cap1 and nvidia-cap2 for CUDA initialization. nvidia-ctk cdi generate (tested up to v1.19.1) does not include these devices in the generated CDI spec. ONNX Runtime (and anything using CUDA) will fail with:

CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
Falling back to ['CPUExecutionProvider'] and retrying.

The container has /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm but is missing /dev/nvidia-caps/. nvidia-smi inside the container will error: Failed to initialize NVML: Unknown Error.

Diagnosis

# 1. Check if caps exist on host (they should)
ls -la /dev/nvidia-caps/
# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)

# 2. Check if caps made it into the container
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
# If "No such file or directory" → CDI spec is missing caps

# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
# If 0 → caps not in the spec

Fix: Manual CDI Spec Patch

The CDI spec lives at /var/run/cdi/nvidia.yaml (Ubuntu 26+) or /etc/cdi/nvidia.yaml. Add caps device nodes to the global containerEdits.deviceNodes section AND each per-device deviceNodes list:

# Under containerEdits.deviceNodes (and each device's deviceNodes):
        - path: /dev/nvidia-caps/nvidia-cap1
          major: 236
          minor: 1
          fileMode: 400
          permissions: r
        - path: /dev/nvidia-caps/nvidia-cap2
          major: 236
          minor: 2
          fileMode: 444
          permissions: r

Use Python to properly insert into the YAML structure (requires sudo — /var/run/cdi/ is root-owned):

# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py

```python
import yaml

caps = [
    {'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
     'fileMode': 400, 'permissions': 'r'},
    {'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
     'fileMode': 444, 'permissions': 'r'},
]

with open('/var/run/cdi/nvidia.yaml', 'r') as f:
    spec = yaml.safe_load(f)

# Global edits
spec['containerEdits']['deviceNodes'].extend(caps)
# Per-device edits
for device in spec.get('devices', []):
    device['containerEdits']['deviceNodes'].extend(caps)

with open('/var/run/cdi/nvidia.yaml', 'w') as f:
    yaml.dump(spec, f, default_flow_style=False, sort_keys=False)

After patching, recreate the container (not just restart — CDI hooks only fire on creation):

docker compose up -d --force-recreate <service>

⚠️ --force-recreate is essential. Without it, docker compose up -d will see the config hasn't changed, report Container <name> Running, and do nothing. The old container (still missing caps) keeps running. Use --force-recreate to force a fresh container that picks up the new CDI spec.

After recreating, verify immediately:

nvidia-smi  # should show GPU processes, not "No running processes found"
docker exec <container> nvidia-smi  # should work (no "Unknown Error")

Note for nvidia-container-toolkit updates

If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g., at boot or service restart). The manual caps entries will be lost. Check after updates and re-apply if needed.