ray
106 lines
3.4 KiB
Markdown
106 lines
3.4 KiB
Markdown
# CDI Caps Device Fix (Driver 580+)
|
|
|
|
## The Problem
|
|
|
|
NVIDIA driver 580+ requires `/dev/nvidia-caps/nvidia-cap1` and `nvidia-cap2` for
|
|
CUDA initialization. `nvidia-ctk cdi generate` (tested up to v1.19.1) **does not
|
|
include these devices** in the generated CDI spec. ONNX Runtime (and anything
|
|
using CUDA) will fail with:
|
|
|
|
```
|
|
CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
|
|
Falling back to ['CPUExecutionProvider'] and retrying.
|
|
```
|
|
|
|
The container has `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm` but is
|
|
missing `/dev/nvidia-caps/`. nvidia-smi inside the container will error:
|
|
`Failed to initialize NVML: Unknown Error`.
|
|
|
|
## Diagnosis
|
|
|
|
```bash
|
|
# 1. Check if caps exist on host (they should)
|
|
ls -la /dev/nvidia-caps/
|
|
# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)
|
|
|
|
# 2. Check if caps made it into the container
|
|
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
|
|
# If "No such file or directory" → CDI spec is missing caps
|
|
|
|
# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
|
|
grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
|
|
# If 0 → caps not in the spec
|
|
```
|
|
|
|
## Fix: Manual CDI Spec Patch
|
|
|
|
The CDI spec lives at `/var/run/cdi/nvidia.yaml` (Ubuntu 26+) or
|
|
`/etc/cdi/nvidia.yaml`. Add caps device nodes to the global
|
|
`containerEdits.deviceNodes` section AND each per-device `deviceNodes` list:
|
|
|
|
```yaml
|
|
# Under containerEdits.deviceNodes (and each device's deviceNodes):
|
|
- path: /dev/nvidia-caps/nvidia-cap1
|
|
major: 236
|
|
minor: 1
|
|
fileMode: 400
|
|
permissions: r
|
|
- path: /dev/nvidia-caps/nvidia-cap2
|
|
major: 236
|
|
minor: 2
|
|
fileMode: 444
|
|
permissions: r
|
|
```
|
|
|
|
Use Python to properly insert into the YAML structure (requires sudo — `/var/run/cdi/` is root-owned):
|
|
|
|
```python
|
|
# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py
|
|
|
|
```python
|
|
import yaml
|
|
|
|
caps = [
|
|
{'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
|
|
'fileMode': 400, 'permissions': 'r'},
|
|
{'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
|
|
'fileMode': 444, 'permissions': 'r'},
|
|
]
|
|
|
|
with open('/var/run/cdi/nvidia.yaml', 'r') as f:
|
|
spec = yaml.safe_load(f)
|
|
|
|
# Global edits
|
|
spec['containerEdits']['deviceNodes'].extend(caps)
|
|
# Per-device edits
|
|
for device in spec.get('devices', []):
|
|
device['containerEdits']['deviceNodes'].extend(caps)
|
|
|
|
with open('/var/run/cdi/nvidia.yaml', 'w') as f:
|
|
yaml.dump(spec, f, default_flow_style=False, sort_keys=False)
|
|
```
|
|
|
|
After patching, **recreate** the container (not just restart — CDI hooks only
|
|
fire on creation):
|
|
|
|
```bash
|
|
docker compose up -d --force-recreate <service>
|
|
```
|
|
|
|
⚠️ **`--force-recreate` is essential.** Without it, `docker compose up -d` will
|
|
see the config hasn't changed, report `Container <name> Running`, and do nothing.
|
|
The old container (still missing caps) keeps running. Use `--force-recreate` to
|
|
force a fresh container that picks up the new CDI spec.
|
|
|
|
After recreating, verify immediately:
|
|
```bash
|
|
nvidia-smi # should show GPU processes, not "No running processes found"
|
|
docker exec <container> nvidia-smi # should work (no "Unknown Error")
|
|
```
|
|
|
|
## Note for nvidia-container-toolkit updates
|
|
|
|
If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g.,
|
|
at boot or service restart). The manual caps entries will be lost. Check after
|
|
updates and re-apply if needed.
|