initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,105 @@
+# CDI Caps Device Fix (Driver 580+)
+
+## The Problem
+
+NVIDIA driver 580+ requires `/dev/nvidia-caps/nvidia-cap1` and `nvidia-cap2` for
+CUDA initialization. `nvidia-ctk cdi generate` (tested up to v1.19.1) **does not
+include these devices** in the generated CDI spec. ONNX Runtime (and anything
+using CUDA) will fail with:
+
+```
+CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
+Falling back to ['CPUExecutionProvider'] and retrying.
+```
+
+The container has `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm` but is
+missing `/dev/nvidia-caps/`. nvidia-smi inside the container will error:
+`Failed to initialize NVML: Unknown Error`.
+
+## Diagnosis
+
+```bash
+# 1. Check if caps exist on host (they should)
+ls -la /dev/nvidia-caps/
+# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)
+
+# 2. Check if caps made it into the container
+docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
+# If "No such file or directory" → CDI spec is missing caps
+
+# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
+grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
+# If 0 → caps not in the spec
+```
+
+## Fix: Manual CDI Spec Patch
+
+The CDI spec lives at `/var/run/cdi/nvidia.yaml` (Ubuntu 26+) or
+`/etc/cdi/nvidia.yaml`. Add caps device nodes to the global
+`containerEdits.deviceNodes` section AND each per-device `deviceNodes` list:
+
+```yaml
+# Under containerEdits.deviceNodes (and each device's deviceNodes):
+        - path: /dev/nvidia-caps/nvidia-cap1
+          major: 236
+          minor: 1
+          fileMode: 400
+          permissions: r
+        - path: /dev/nvidia-caps/nvidia-cap2
+          major: 236
+          minor: 2
+          fileMode: 444
+          permissions: r
+```
+
+Use Python to properly insert into the YAML structure (requires sudo — `/var/run/cdi/` is root-owned):
+
+```python
+# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py
+
+```python
+import yaml
+
+caps = [
+    {'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
+     'fileMode': 400, 'permissions': 'r'},
+    {'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
+     'fileMode': 444, 'permissions': 'r'},
+]
+
+with open('/var/run/cdi/nvidia.yaml', 'r') as f:
+    spec = yaml.safe_load(f)
+
+# Global edits
+spec['containerEdits']['deviceNodes'].extend(caps)
+# Per-device edits
+for device in spec.get('devices', []):
+    device['containerEdits']['deviceNodes'].extend(caps)
+
+with open('/var/run/cdi/nvidia.yaml', 'w') as f:
+    yaml.dump(spec, f, default_flow_style=False, sort_keys=False)
+```
+
+After patching, **recreate** the container (not just restart — CDI hooks only
+fire on creation):
+
+```bash
+docker compose up -d --force-recreate <service>
+```
+
+⚠️  **`--force-recreate` is essential.** Without it, `docker compose up -d` will
+see the config hasn't changed, report `Container <name> Running`, and do nothing.
+The old container (still missing caps) keeps running. Use `--force-recreate` to
+force a fresh container that picks up the new CDI spec.
+
+After recreating, verify immediately:
+```bash
+nvidia-smi  # should show GPU processes, not "No running processes found"
+docker exec <container> nvidia-smi  # should work (no "Unknown Error")
+```
+
+## Note for nvidia-container-toolkit updates
+
+If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g.,
+at boot or service restart). The manual caps entries will be lost. Check after
+updates and re-apply if needed.