# CDI Caps Device Fix (Driver 580+)

## The Problem

NVIDIA driver 580+ requires `/dev/nvidia-caps/nvidia-cap1` and `nvidia-cap2` for
CUDA initialization. `nvidia-ctk cdi generate` (tested up to v1.19.1) **does not
include these devices** in the generated CDI spec. ONNX Runtime (and anything
using CUDA) will fail with:

```
CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
Falling back to ['CPUExecutionProvider'] and retrying.
```

The container has `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm` but is
missing `/dev/nvidia-caps/`. nvidia-smi inside the container will error:
`Failed to initialize NVML: Unknown Error`.

## Diagnosis

```bash
# 1. Check if caps exist on host (they should)
ls -la /dev/nvidia-caps/
# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)

# 2. Check if caps made it into the container
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
# If "No such file or directory" → CDI spec is missing caps

# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
# If 0 → caps not in the spec
```

## Fix: Manual CDI Spec Patch

The CDI spec lives at `/var/run/cdi/nvidia.yaml` (Ubuntu 26+) or
`/etc/cdi/nvidia.yaml`. Add caps device nodes to the global
`containerEdits.deviceNodes` section AND each per-device `deviceNodes` list:

```yaml
# Under containerEdits.deviceNodes (and each device's deviceNodes):
        - path: /dev/nvidia-caps/nvidia-cap1
          major: 236
          minor: 1
          fileMode: 400
          permissions: r
        - path: /dev/nvidia-caps/nvidia-cap2
          major: 236
          minor: 2
          fileMode: 444
          permissions: r
```

Use Python to properly insert into the YAML structure (requires sudo — `/var/run/cdi/` is root-owned):

```python
# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py

```python
import yaml

caps = [
    {'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
     'fileMode': 400, 'permissions': 'r'},
    {'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
     'fileMode': 444, 'permissions': 'r'},
]

with open('/var/run/cdi/nvidia.yaml', 'r') as f:
    spec = yaml.safe_load(f)

# Global edits
spec['containerEdits']['deviceNodes'].extend(caps)
# Per-device edits
for device in spec.get('devices', []):
    device['containerEdits']['deviceNodes'].extend(caps)

with open('/var/run/cdi/nvidia.yaml', 'w') as f:
    yaml.dump(spec, f, default_flow_style=False, sort_keys=False)
```

After patching, **recreate** the container (not just restart — CDI hooks only
fire on creation):

```bash
docker compose up -d --force-recreate <service>
```

⚠️  **`--force-recreate` is essential.** Without it, `docker compose up -d` will
see the config hasn't changed, report `Container <name> Running`, and do nothing.
The old container (still missing caps) keeps running. Use `--force-recreate` to
force a fresh container that picks up the new CDI spec.

After recreating, verify immediately:
```bash
nvidia-smi  # should show GPU processes, not "No running processes found"
docker exec <container> nvidia-smi  # should work (no "Unknown Error")
```

## Note for nvidia-container-toolkit updates

If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g.,
at boot or service restart). The manual caps entries will be lost. Check after
updates and re-apply if needed.