initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,162 @@
+---
+name: docker-gpu-acceleration
+description: Set up, verify, and troubleshoot NVIDIA GPU acceleration for Docker containers running ML/AI services (Immich ML, ONNX models, LLMs, etc.)
+category: self-hosting
+---
+
+# Docker GPU Acceleration
+
+Set up NVIDIA GPU access in Docker containers for ML/AI workloads and debug when it isn't working.
+
+## When to use
+
+- User wants GPU acceleration in Docker for Immich ML, LLM serving, or ONNX inference
+- `nvidia-smi` shows 0% util / 11 MiB / no processes — GPU idle when it should be working
+- Container repeatedly fails to load ML models (download→fail→clear→retry loop)
+- Image uses `-cuda` suffix but GPU isn't actually being used
+
+## Steps
+
+### 1. Verify host GPU is functional
+
+```bash
+nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv,noheader
+```
+
+**Idle baseline**: ~11 MiB memory, 0% util, P8 power state
+**Active**: >100 MiB memory, >0% util, P0 power state
+
+### 2. Check Docker nvidia runtime is available
+
+```bash
+docker info | grep -i "runtimes"
+```
+
+Must show `nvidia` in the list. If not, install `nvidia-container-toolkit`:
+
+```bash
+apt install nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo systemctl restart docker
+```
+
+### 3. Add GPU access to docker-compose.yml
+
+Add to the service that needs the GPU:
+
+```yaml
+deploy:
+  resources:
+    reservations:
+      devices:
+        - driver: nvidia
+          count: all
+          capabilities: [gpu]
+```
+
+Then recreate: `docker compose up -d <service>`
+
+### 4. Verify GPU access inside container
+
+```bash
+# Check NVIDIA devices
+docker exec <container> ls -la /dev | grep nvidia
+
+# Should show: nvidia0, nvidiactl, nvidia-uvm, nvidia-caps (driver 580+)
+
+# Check caps specifically (driver 580+ requirement)
+docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
+# If "No such file or directory" → CDI spec is missing caps. See references/cdi-caps-fix.md
+
+# Check ONNX Runtime sees GPU
+docker exec <container> python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"
+
+# Should show: GPU and ['CUDAExecutionProvider', 'TensorrtExecutionProvider', 'CPUExecutionProvider']
+```
+
+### 5. Test actual GPU inference
+
+Create a minimal ONNX model and run it with CUDAExecutionProvider to verify the GPU executes work, not just reports as available.
+
+### 6. Pre-cache models to avoid download loops
+
+If the service downloads→fails→clears→retries in a loop, the model cache is empty. Download manually:
+
+```python
+from huggingface_hub import snapshot_download
+result = snapshot_download(
+    "immich-app/<model_name>",
+    cache_dir="/cache/<model_task>/<model_name>",
+    local_dir="/cache/<model_task>/<model_name>",
+    ignore_patterns=["*.armnn", "*.rknn"],
+)
+```
+
+Then restart the container — it picks up cached models and loads immediately.
+
+### 7. Monitor GPU utilization
+
+```bash
+nvidia-smi  # snapshot view
+nvtop       # live TUI (install with apt install nvtop)
+```
+
+### 8. (Optional) Cockpit Web UI Dashboard
+
+Create a custom Cockpit package that shows live GPU metrics in the web UI
+sidebar at `https://<server>:9090`:
+
+```bash
+sudo mkdir -p /usr/share/cockpit/nvidia-gpu
+```
+
+**⚠️ Important:** `cockpit.script()` (run nvidia-smi directly) silently fails
+on Cockpit v314+ (Ubuntu 26+). The **recommended approach** is a systemd
+service that writes GPU data to `/run/*.txt` files, then the Cockpit page
+reads them via `cockpit.file().read()`.
+
+Follow the full instructions in `references/cockpit-gpu-dashboard.md` — it
+documents both approaches with the correct working procedure.
+
+```bash
+sudo systemctl restart cockpit
+```
+
+The page displays model, driver, CUDA version, temperature, utilization,
+memory bar, power draw, and running processes — refreshing every 5 seconds.
+
+### 9. Verify GPU Dashboard
+
+After setup, run the verification script:
+
+```bash
+~/.hermes/skills/self-hosting/docker-gpu-acceleration/scripts/verify-cockpit-gpu.sh
+```
+
+This checks the systemd service, data files, Cockpit package installation,
+nvidia-smi accessibility, and Cockpit service health.
+
+## Pitfalls
+
+- **Compose shows `Runtime: runc` and `DeviceRequests: null`** — GPU was never configured. Add `deploy.resources.reservations.devices` to the service.
+- **ONNX Runtime reports "no CUDA-capable device is detected" / nvidia-smi fails inside container** — Likely missing `/dev/nvidia-caps/`. Check `docker exec <container> ls /dev/nvidia-caps/`. If absent despite host having them, the CDI spec (at `/var/run/cdi/nvidia.yaml`) is missing caps device entries. See `references/cdi-caps-fix.md` for the fix. Driver 580+ requires caps for CUDA initialization.
+- **Model download→fail→clear→retry loop** — Cache directory is empty. Pre-download models, then restart container.
+- **EHOSTUNREACH between containers on same bridge** — Docker bridge networking glitch. `docker compose restart database redis server` to fix.
+- **Model download returns 401** — Don't use raw HuggingFace URL. Use `huggingface_hub.snapshot_download()` Python API, which handles auth correctly.
+- **Container CUDA 12.2 on host CUDA 13.0** — Usually fine (CUDA backward-compatible within major versions), but if models fail to load, check the exact error.
+- **4 GB VRAM limit** — Some GPU-inference stacks need all models loaded simultaneously. If VRAM fills, consider `MACHINE_LEARNING_MODEL_ARENA=true` (loads one model at a time).
+- **nvtop not available in apt** — Build from source at `https://github.com/Syllo/nvtop`.
+
+## Verification checklist
+
+- [ ] `nvidia-smi` shows python process using GPU memory
+- [ ] Service container logs show successful model loading (no "Failed to load" warnings)
+- [ ] Service health check passes
+- [ ] Job queues show active / waiting items (not stuck at 0)
+- [ ] GPU temp rises above idle (28-30°C → 40-65°C under load)
+
+## Related
+
+- `references/immich-ml-gpu-paths.md` — Exact cache paths and model names for Immich ML
+- `references/cockpit-gpu-dashboard.md` — Cockpit web UI GPU dashboard package
+- `references/cdi-caps-fix.md` — CDI spec missing `/dev/nvidia-caps/` on driver 580+ (CUDA fails, ONNX falls back to CPU)
@@ -0,0 +1,105 @@
+# CDI Caps Device Fix (Driver 580+)
+
+## The Problem
+
+NVIDIA driver 580+ requires `/dev/nvidia-caps/nvidia-cap1` and `nvidia-cap2` for
+CUDA initialization. `nvidia-ctk cdi generate` (tested up to v1.19.1) **does not
+include these devices** in the generated CDI spec. ONNX Runtime (and anything
+using CUDA) will fail with:
+
+```
+CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
+Falling back to ['CPUExecutionProvider'] and retrying.
+```
+
+The container has `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm` but is
+missing `/dev/nvidia-caps/`. nvidia-smi inside the container will error:
+`Failed to initialize NVML: Unknown Error`.
+
+## Diagnosis
+
+```bash
+# 1. Check if caps exist on host (they should)
+ls -la /dev/nvidia-caps/
+# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)
+
+# 2. Check if caps made it into the container
+docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
+# If "No such file or directory" → CDI spec is missing caps
+
+# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
+grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
+# If 0 → caps not in the spec
+```
+
+## Fix: Manual CDI Spec Patch
+
+The CDI spec lives at `/var/run/cdi/nvidia.yaml` (Ubuntu 26+) or
+`/etc/cdi/nvidia.yaml`. Add caps device nodes to the global
+`containerEdits.deviceNodes` section AND each per-device `deviceNodes` list:
+
+```yaml
+# Under containerEdits.deviceNodes (and each device's deviceNodes):
+        - path: /dev/nvidia-caps/nvidia-cap1
+          major: 236
+          minor: 1
+          fileMode: 400
+          permissions: r
+        - path: /dev/nvidia-caps/nvidia-cap2
+          major: 236
+          minor: 2
+          fileMode: 444
+          permissions: r
+```
+
+Use Python to properly insert into the YAML structure (requires sudo — `/var/run/cdi/` is root-owned):
+
+```python
+# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py
+
+```python
+import yaml
+
+caps = [
+    {'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
+     'fileMode': 400, 'permissions': 'r'},
+    {'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
+     'fileMode': 444, 'permissions': 'r'},
+]
+
+with open('/var/run/cdi/nvidia.yaml', 'r') as f:
+    spec = yaml.safe_load(f)
+
+# Global edits
+spec['containerEdits']['deviceNodes'].extend(caps)
+# Per-device edits
+for device in spec.get('devices', []):
+    device['containerEdits']['deviceNodes'].extend(caps)
+
+with open('/var/run/cdi/nvidia.yaml', 'w') as f:
+    yaml.dump(spec, f, default_flow_style=False, sort_keys=False)
+```
+
+After patching, **recreate** the container (not just restart — CDI hooks only
+fire on creation):
+
+```bash
+docker compose up -d --force-recreate <service>
+```
+
+⚠️  **`--force-recreate` is essential.** Without it, `docker compose up -d` will
+see the config hasn't changed, report `Container <name> Running`, and do nothing.
+The old container (still missing caps) keeps running. Use `--force-recreate` to
+force a fresh container that picks up the new CDI spec.
+
+After recreating, verify immediately:
+```bash
+nvidia-smi  # should show GPU processes, not "No running processes found"
+docker exec <container> nvidia-smi  # should work (no "Unknown Error")
+```
+
+## Note for nvidia-container-toolkit updates
+
+If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g.,
+at boot or service restart). The manual caps entries will be lost. Check after
+updates and re-apply if needed.
@@ -0,0 +1,159 @@
+# Cockpit NVIDIA GPU Dashboard
+
+> Custom Cockpit package that displays live GPU stats (model, driver, CUDA
+> version, temperature, utilization, memory, processes) in the web UI sidebar
+> on port 9090.
+
+## When to use
+
+- User wants GPU temperature / utilization / memory visible in Cockpit
+- User prefers a web UI over terminal `nvidia-smi` or `nvtop`
+
+## Two approaches
+
+`cockpit.script()` (run nvidia-smi directly) **silently fails on some Cockpit
+versions** (Ubuntu 26+, Cockpit ~314+). The **systemd + file-read** approach
+is more reliable and preferred.
+
+---
+
+## Approach A (recommended): systemd service + file-read
+
+Write GPU data to files periodically via a background service, then read them
+from the Cockpit page with `cockpit.file().read()`. Avoids all PATH and
+permission issues with running nvidia-smi from the Cockpit bridge.
+
+### 1. Create the data-collection script
+
+**`/usr/local/bin/nvidia-gpu-monitor.sh`:**
+
+```bash
+#!/bin/bash
+while true; do
+  nvidia-smi --query-gpu=name,driver_version,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw,pstate,fan.speed --format=csv,noheader,nounits 2>/dev/null | head -1 > /run/nvidia-gpu.txt
+  nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits 2>/dev/null > /run/nvidia-processes.txt
+  nvidia-smi 2>/dev/null | grep "CUDA Version" | sed 's/.*CUDA Version: //' | head -1 > /run/nvidia-cuda.txt
+  sleep 5
+done
+```
+
+Make executable: `sudo chmod +x /usr/local/bin/nvidia-gpu-monitor.sh`
+
+### 2. Create systemd service
+
+**`/etc/systemd/system/nvidia-gpu-monitor.service`:**
+
+```ini
+[Unit]
+Description=NVIDIA GPU Monitor - writes stats to /run
+After=nvidia-persistenced.service
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/nvidia-gpu-monitor.sh
+Restart=always
+RestartSec=3
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Enable and start:
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now nvidia-gpu-monitor
+```
+
+### 3. Create the Cockpit package
+
+**`/usr/share/cockpit/nvidia-gpu/manifest.json`:**
+
+```json
+{
+    "version": 1,
+    "requires": { "cockpit": "260" },
+    "menu": {
+        "index": { "label": "NVIDIA GPU", "order": 90 }
+    },
+    "content-security-policy": "default-src 'self'; script-src 'unsafe-inline'"
+}
+```
+
+**`/usr/share/cockpit/nvidia-gpu/index.html`:**
+
+The page uses `cockpit.file(path).read()` to read `/run/nvidia-gpu.txt`,
+`/run/nvidia-processes.txt`, and `/run/nvidia-cuda.txt`. Each file contains
+a single line of comma-separated values written by the systemd service.
+
+Key implementation:
+```javascript
+function readFile(path) { return cockpit.file(path).read(); }
+
+function update() {
+  readFile('/run/nvidia-gpu.txt').done(function(data) {
+    if (!data || !data.trim()) { /* show error */ return; }
+    var parts = data.trim().split(', ');
+    // parts[0] = name, [1] = driver, [2] = temp, [3] = util,
+    // [4] = mem_used, [5] = mem_total, [6] = power, [7] = pstate, [8] = fan
+    // ... render cards ...
+  });
+
+  readFile('/run/nvidia-cuda.txt').done(function(d) { /* show CUDA version */ });
+  readFile('/run/nvidia-processes.txt').done(function(d) { /* render process table */ });
+}
+setInterval(update, 5000);
+```
+
+### 4. Install and restart Cockpit
+
+```bash
+sudo mkdir -p /usr/share/cockpit/nvidia-gpu
+# Write manifest.json and index.html (use sudo tee or sudo cp)
+sudo chmod 644 /usr/share/cockpit/nvidia-gpu/*
+sudo systemctl restart cockpit
+```
+
+### 5. Verify
+
+1. Open `https://<server>:9090` in browser
+2. Log in with server credentials
+3. Click **NVIDIA GPU** in sidebar — live metrics refresh every 5s
+
+---
+
+## Approach B (fallback): cockpit.script() directly
+
+Use when Approach A isn't practical (no systemd, containerized Cockpit, etc.):
+
+```javascript
+cockpit.script("/usr/bin/nvidia-smi --query-gpu=... --format=csv,noheader,nounits", null, {superuser: 'try'})
+  .done(function(data) { /* parse and render */ })
+  .fail(function(err) { /* show error */ });
+```
+
+**Caveats (why Approach A is preferred):**
+- `cockpit.script()` may silently hang on Cockpit v314+ (Ubuntu 26+)
+- PATH may not include `/usr/bin/` in non-interactive bridge sessions
+- The `.fail()` handler may catch but not always fire for certain errors
+- Debugging is harder — the error object from Cockpit bridge is opaque
+
+If using this approach, use the full path `/usr/bin/nvidia-smi` and avoid
+`{superuser: 'try'}` unless nvidia-smi requires root on the target system.
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely Cause | Fix |
+|---------|-------------|-----|
+| "No NVIDIA GPU detected" | Can't read data files or run nvidia-smi | Check `/run/nvidia-gpu.txt` exists and is populated. Restart service: `sudo systemctl restart nvidia-gpu-monitor` |
+| Page shows stale data | systemd service stopped or crashed | `sudo systemctl status nvidia-gpu-monitor` — restart if dead |
+| Power shows `[N/A]` | GTX 1050 Ti doesn't report power draw | Normal — handle gracefully in display |
+| Page loads blank | CSP blocking inline script | Add `'unsafe-inline'` to manifest CSP |
+| "No GPU processes" when Immich is processing | Jobs may be between batches | Check `nvidia-smi` directly — should show python process |
+| Ctrl+Shift+R needed after updates | Cockpit aggressively caches package content | Hard-refresh or open in private/incognito tab |
+
+## Related
+
+- Install `nvtop` for terminal TUI monitor: `apt install nvtop` (or build from https://github.com/Syllo/nvtop)
+- GPU acceleration setup: see the parent `docker-gpu-acceleration` skill
@@ -0,0 +1,103 @@
+# Immich ML GPU Cache Paths and Model Names
+
+Known working for Immich v2.7.5.
+
+## Model Sources
+
+All models hosted on HuggingFace under `immich-app/` org.
+
+| Model Name | HuggingFace Repo | Files | Size |
+|---|---|---|---|
+| `ViT-B-32__openai` | `immich-app/ViT-B-32__openai` | visual/model.onnx (335 MB), textual/model.onnx (254 MB) | ~590 MB total |
+| `buffalo_l` | `immich-app/buffalo_l` | detection/model.onnx (16 MB), recognition/model.onnx (166 MB) | ~182 MB total |
+
+## Cache Directory Structure
+
+Cache root: `/cache` (from `MACHINE_LEARNING_CACHE_FOLDER` env var).
+
+### CLIP (smart search)
+
+| Field | Value |
+|---|---|
+| Model task | `SEARCH` → `"clip"` |
+| Model type | `VISUAL` → `"visual"`, `TEXTUAL` → `"textual"` |
+| Cache dir | `/cache/clip/ViT-B-32__openai` |
+| Model dir (visual) | `/cache/clip/ViT-B-32__openai/visual` |
+| Model path (visual) | `/cache/clip/ViT-B-32__openai/visual/model.onnx` |
+| Model dir (textual) | `/cache/clip/ViT-B-32__openai/textual` |
+| Model path (textual) | `/cache/clip/ViT-B-32__openai/textual/model.onnx` |
+
+### Face detection & recognition
+
+| Field | Value |
+|---|---|
+| Model task | `FACIAL_RECOGNITION` → `"facial-recognition"` |
+| Cache dir | `/cache/facial-recognition/buffalo_l` |
+| Model dir (detection) | `/cache/facial-recognition/buffalo_l/detection` |
+| Model path (detection) | `/cache/facial-recognition/buffalo_l/detection/model.onnx` |
+| Model dir (recognition) | `/cache/facial-recognition/buffalo_l/recognition` |
+| Model path (recognition) | `/cache/facial-recognition/buffalo_l/recognition/model.onnx` |
+
+## Code Derivation
+
+Cache paths follow this pattern:
+
+```python
+from immich_ml.config import settings
+# settings.cache_folder = "/cache"
+# model_task.value = one of: "clip", "facial-recognition", "ocr"
+# model_name = e.g. "ViT-B-32__openai" or "buffalo_l"
+# model_type.value = one of: "visual", "textual", "detection", "recognition"
+
+cache_dir = settings.cache_folder / model_task.value / model_name
+model_dir = cache_dir / model_type.value
+model_path = model_dir / "model.onnx"
+```
+
+## Pre-download command
+
+```python
+from huggingface_hub import snapshot_download
+import os
+
+# For CLIP
+os.makedirs("/cache/clip/ViT-B-32__openai", exist_ok=True)
+snapshot_download(
+    "immich-app/ViT-B-32__openai",
+    cache_dir="/cache/clip/ViT-B-32__openai",
+    local_dir="/cache/clip/ViT-B-32__openai",
+    ignore_patterns=["*.armnn", "*.rknn"],
+)
+
+# For face detection/recognition
+os.makedirs("/cache/facial-recognition/buffalo_l", exist_ok=True)
+snapshot_download(
+    "immich-app/buffalo_l",
+    cache_dir="/cache/facial-recognition/buffalo_l",
+    local_dir="/cache/facial-recognition/buffalo_l",
+    ignore_patterns=["*.armnn", "*.rknn"],
+)
+```
+
+## docker-compose GPU config
+
+Required addition to `immich-machine-learning` service:
+
+```yaml
+deploy:
+  resources:
+    reservations:
+      devices:
+        - driver: nvidia
+          count: all
+          capabilities: [gpu]
+```
+
+## Environment variables
+
+| Variable | Value | Effect |
+|---|---|---|
+| `DEVICE` | `cuda` | Use CUDA (set by -cuda image) |
+| `MACHINE_LEARNING_MODEL_ARENA` | `false` | Load all models at once. Set `true` to load one at a time (saves VRAM but slower switching) |
+| `MACHINE_LEARNING_DEVICE_ID` | `0` | GPU device index |
+| `MACHINE_LEARNING_CACHE_FOLDER` | `/cache` | Where models are stored |
@@ -0,0 +1,65 @@
+#!/bin/bash
+# Verify Cockpit NVIDIA GPU dashboard health
+# Returns non-zero exit code if anything is wrong.
+
+FAIL=0
+
+echo "=== Checking systemd service ==="
+if systemctl is-active --quiet nvidia-gpu-monitor; then
+    echo "  [PASS] nvidia-gpu-monitor service is running"
+else
+    echo "  [FAIL] nvidia-gpu-monitor service is NOT running"
+    FAIL=1
+fi
+
+echo ""
+echo "=== Checking data files ==="
+for f in /run/nvidia-gpu.txt /run/nvidia-processes.txt /run/nvidia-cuda.txt; do
+    if [ -f "$f" ]; then
+        CONTENT=$(head -1 "$f")
+        echo "  [PASS] $f: $CONTENT"
+    else
+        echo "  [FAIL] $f does not exist"
+        FAIL=1
+    fi
+done
+
+echo ""
+echo "=== Checking Cockpit package ==="
+MANIFEST="/usr/share/cockpit/nvidia-gpu/manifest.json"
+INDEX="/usr/share/cockpit/nvidia-gpu/index.html"
+if [ -f "$MANIFEST" ] && [ -f "$INDEX" ]; then
+    echo "  [PASS] Cockpit nvidia-gpu package installed"
+    echo "  Manifest: $(wc -c < "$MANIFEST") bytes"
+    echo "  Index:    $(wc -c < "$INDEX") bytes"
+else
+    echo "  [FAIL] Cockpit package files missing"
+    FAIL=1
+fi
+
+echo ""
+echo "=== Checking GPU accessibility ==="
+GPU_LINE=$(nvidia-smi --query-gpu=name --format=csv,noheader 2>/dev/null | head -1)
+if [ -n "$GPU_LINE" ]; then
+    echo "  [PASS] GPU detected: $GPU_LINE"
+else
+    echo "  [FAIL] nvidia-smi not working"
+    FAIL=1
+fi
+
+echo ""
+echo "=== Cockpit service ==="
+if systemctl is-active --quiet cockpit; then
+    echo "  [PASS] Cockpit running on port 9090"
+else
+    echo "  [FAIL] Cockpit is not running"
+    FAIL=1
+fi
+
+echo ""
+if [ "$FAIL" -eq 0 ]; then
+    echo "All checks passed. GPU dashboard should be functional."
+else
+    echo "Some checks FAILED. See above."
+fi
+exit $FAIL