initial commit
This commit is contained in:
ray
@@ -0,0 +1,105 @@
|
||||
# CDI Caps Device Fix (Driver 580+)
|
||||
|
||||
## The Problem
|
||||
|
||||
NVIDIA driver 580+ requires `/dev/nvidia-caps/nvidia-cap1` and `nvidia-cap2` for
|
||||
CUDA initialization. `nvidia-ctk cdi generate` (tested up to v1.19.1) **does not
|
||||
include these devices** in the generated CDI spec. ONNX Runtime (and anything
|
||||
using CUDA) will fail with:
|
||||
|
||||
```
|
||||
CUDA failure 100: no CUDA-capable device is detected ; GPU=-1
|
||||
Falling back to ['CPUExecutionProvider'] and retrying.
|
||||
```
|
||||
|
||||
The container has `/dev/nvidia0`, `/dev/nvidiactl`, `/dev/nvidia-uvm` but is
|
||||
missing `/dev/nvidia-caps/`. nvidia-smi inside the container will error:
|
||||
`Failed to initialize NVML: Unknown Error`.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
```bash
|
||||
# 1. Check if caps exist on host (they should)
|
||||
ls -la /dev/nvidia-caps/
|
||||
# Output should show: nvidia-cap1 (major 236, minor 1), nvidia-cap2 (236, 2)
|
||||
|
||||
# 2. Check if caps made it into the container
|
||||
docker exec <container> ls -la /dev/nvidia-caps/ 2>&1
|
||||
# If "No such file or directory" → CDI spec is missing caps
|
||||
|
||||
# 3. Check the CDI spec (at /var/run/cdi/nvidia.yaml or /etc/cdi/nvidia.yaml)
|
||||
grep -c 'nvidia-cap' /var/run/cdi/nvidia.yaml
|
||||
# If 0 → caps not in the spec
|
||||
```
|
||||
|
||||
## Fix: Manual CDI Spec Patch
|
||||
|
||||
The CDI spec lives at `/var/run/cdi/nvidia.yaml` (Ubuntu 26+) or
|
||||
`/etc/cdi/nvidia.yaml`. Add caps device nodes to the global
|
||||
`containerEdits.deviceNodes` section AND each per-device `deviceNodes` list:
|
||||
|
||||
```yaml
|
||||
# Under containerEdits.deviceNodes (and each device's deviceNodes):
|
||||
- path: /dev/nvidia-caps/nvidia-cap1
|
||||
major: 236
|
||||
minor: 1
|
||||
fileMode: 400
|
||||
permissions: r
|
||||
- path: /dev/nvidia-caps/nvidia-cap2
|
||||
major: 236
|
||||
minor: 2
|
||||
fileMode: 444
|
||||
permissions: r
|
||||
```
|
||||
|
||||
Use Python to properly insert into the YAML structure (requires sudo — `/var/run/cdi/` is root-owned):
|
||||
|
||||
```python
|
||||
# Save to a temp file, then run: sudo python3 /tmp/fix-cdi.py
|
||||
|
||||
```python
|
||||
import yaml
|
||||
|
||||
caps = [
|
||||
{'path': '/dev/nvidia-caps/nvidia-cap1', 'major': 236, 'minor': 1,
|
||||
'fileMode': 400, 'permissions': 'r'},
|
||||
{'path': '/dev/nvidia-caps/nvidia-cap2', 'major': 236, 'minor': 2,
|
||||
'fileMode': 444, 'permissions': 'r'},
|
||||
]
|
||||
|
||||
with open('/var/run/cdi/nvidia.yaml', 'r') as f:
|
||||
spec = yaml.safe_load(f)
|
||||
|
||||
# Global edits
|
||||
spec['containerEdits']['deviceNodes'].extend(caps)
|
||||
# Per-device edits
|
||||
for device in spec.get('devices', []):
|
||||
device['containerEdits']['deviceNodes'].extend(caps)
|
||||
|
||||
with open('/var/run/cdi/nvidia.yaml', 'w') as f:
|
||||
yaml.dump(spec, f, default_flow_style=False, sort_keys=False)
|
||||
```
|
||||
|
||||
After patching, **recreate** the container (not just restart — CDI hooks only
|
||||
fire on creation):
|
||||
|
||||
```bash
|
||||
docker compose up -d --force-recreate <service>
|
||||
```
|
||||
|
||||
⚠️ **`--force-recreate` is essential.** Without it, `docker compose up -d` will
|
||||
see the config hasn't changed, report `Container <name> Running`, and do nothing.
|
||||
The old container (still missing caps) keeps running. Use `--force-recreate` to
|
||||
force a fresh container that picks up the new CDI spec.
|
||||
|
||||
After recreating, verify immediately:
|
||||
```bash
|
||||
nvidia-smi # should show GPU processes, not "No running processes found"
|
||||
docker exec <container> nvidia-smi # should work (no "Unknown Error")
|
||||
```
|
||||
|
||||
## Note for nvidia-container-toolkit updates
|
||||
|
||||
If nvidia-container-toolkit is updated, the CDI spec may be regenerated (e.g.,
|
||||
at boot or service restart). The manual caps entries will be lost. Check after
|
||||
updates and re-apply if needed.
|
||||
@@ -0,0 +1,159 @@
|
||||
# Cockpit NVIDIA GPU Dashboard
|
||||
|
||||
> Custom Cockpit package that displays live GPU stats (model, driver, CUDA
|
||||
> version, temperature, utilization, memory, processes) in the web UI sidebar
|
||||
> on port 9090.
|
||||
|
||||
## When to use
|
||||
|
||||
- User wants GPU temperature / utilization / memory visible in Cockpit
|
||||
- User prefers a web UI over terminal `nvidia-smi` or `nvtop`
|
||||
|
||||
## Two approaches
|
||||
|
||||
`cockpit.script()` (run nvidia-smi directly) **silently fails on some Cockpit
|
||||
versions** (Ubuntu 26+, Cockpit ~314+). The **systemd + file-read** approach
|
||||
is more reliable and preferred.
|
||||
|
||||
---
|
||||
|
||||
## Approach A (recommended): systemd service + file-read
|
||||
|
||||
Write GPU data to files periodically via a background service, then read them
|
||||
from the Cockpit page with `cockpit.file().read()`. Avoids all PATH and
|
||||
permission issues with running nvidia-smi from the Cockpit bridge.
|
||||
|
||||
### 1. Create the data-collection script
|
||||
|
||||
**`/usr/local/bin/nvidia-gpu-monitor.sh`:**
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
while true; do
|
||||
nvidia-smi --query-gpu=name,driver_version,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw,pstate,fan.speed --format=csv,noheader,nounits 2>/dev/null | head -1 > /run/nvidia-gpu.txt
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits 2>/dev/null > /run/nvidia-processes.txt
|
||||
nvidia-smi 2>/dev/null | grep "CUDA Version" | sed 's/.*CUDA Version: //' | head -1 > /run/nvidia-cuda.txt
|
||||
sleep 5
|
||||
done
|
||||
```
|
||||
|
||||
Make executable: `sudo chmod +x /usr/local/bin/nvidia-gpu-monitor.sh`
|
||||
|
||||
### 2. Create systemd service
|
||||
|
||||
**`/etc/systemd/system/nvidia-gpu-monitor.service`:**
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=NVIDIA GPU Monitor - writes stats to /run
|
||||
After=nvidia-persistenced.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/nvidia-gpu-monitor.sh
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable and start:
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now nvidia-gpu-monitor
|
||||
```
|
||||
|
||||
### 3. Create the Cockpit package
|
||||
|
||||
**`/usr/share/cockpit/nvidia-gpu/manifest.json`:**
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"requires": { "cockpit": "260" },
|
||||
"menu": {
|
||||
"index": { "label": "NVIDIA GPU", "order": 90 }
|
||||
},
|
||||
"content-security-policy": "default-src 'self'; script-src 'unsafe-inline'"
|
||||
}
|
||||
```
|
||||
|
||||
**`/usr/share/cockpit/nvidia-gpu/index.html`:**
|
||||
|
||||
The page uses `cockpit.file(path).read()` to read `/run/nvidia-gpu.txt`,
|
||||
`/run/nvidia-processes.txt`, and `/run/nvidia-cuda.txt`. Each file contains
|
||||
a single line of comma-separated values written by the systemd service.
|
||||
|
||||
Key implementation:
|
||||
```javascript
|
||||
function readFile(path) { return cockpit.file(path).read(); }
|
||||
|
||||
function update() {
|
||||
readFile('/run/nvidia-gpu.txt').done(function(data) {
|
||||
if (!data || !data.trim()) { /* show error */ return; }
|
||||
var parts = data.trim().split(', ');
|
||||
// parts[0] = name, [1] = driver, [2] = temp, [3] = util,
|
||||
// [4] = mem_used, [5] = mem_total, [6] = power, [7] = pstate, [8] = fan
|
||||
// ... render cards ...
|
||||
});
|
||||
|
||||
readFile('/run/nvidia-cuda.txt').done(function(d) { /* show CUDA version */ });
|
||||
readFile('/run/nvidia-processes.txt').done(function(d) { /* render process table */ });
|
||||
}
|
||||
setInterval(update, 5000);
|
||||
```
|
||||
|
||||
### 4. Install and restart Cockpit
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /usr/share/cockpit/nvidia-gpu
|
||||
# Write manifest.json and index.html (use sudo tee or sudo cp)
|
||||
sudo chmod 644 /usr/share/cockpit/nvidia-gpu/*
|
||||
sudo systemctl restart cockpit
|
||||
```
|
||||
|
||||
### 5. Verify
|
||||
|
||||
1. Open `https://<server>:9090` in browser
|
||||
2. Log in with server credentials
|
||||
3. Click **NVIDIA GPU** in sidebar — live metrics refresh every 5s
|
||||
|
||||
---
|
||||
|
||||
## Approach B (fallback): cockpit.script() directly
|
||||
|
||||
Use when Approach A isn't practical (no systemd, containerized Cockpit, etc.):
|
||||
|
||||
```javascript
|
||||
cockpit.script("/usr/bin/nvidia-smi --query-gpu=... --format=csv,noheader,nounits", null, {superuser: 'try'})
|
||||
.done(function(data) { /* parse and render */ })
|
||||
.fail(function(err) { /* show error */ });
|
||||
```
|
||||
|
||||
**Caveats (why Approach A is preferred):**
|
||||
- `cockpit.script()` may silently hang on Cockpit v314+ (Ubuntu 26+)
|
||||
- PATH may not include `/usr/bin/` in non-interactive bridge sessions
|
||||
- The `.fail()` handler may catch but not always fire for certain errors
|
||||
- Debugging is harder — the error object from Cockpit bridge is opaque
|
||||
|
||||
If using this approach, use the full path `/usr/bin/nvidia-smi` and avoid
|
||||
`{superuser: 'try'}` unless nvidia-smi requires root on the target system.
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
| Symptom | Likely Cause | Fix |
|
||||
|---------|-------------|-----|
|
||||
| "No NVIDIA GPU detected" | Can't read data files or run nvidia-smi | Check `/run/nvidia-gpu.txt` exists and is populated. Restart service: `sudo systemctl restart nvidia-gpu-monitor` |
|
||||
| Page shows stale data | systemd service stopped or crashed | `sudo systemctl status nvidia-gpu-monitor` — restart if dead |
|
||||
| Power shows `[N/A]` | GTX 1050 Ti doesn't report power draw | Normal — handle gracefully in display |
|
||||
| Page loads blank | CSP blocking inline script | Add `'unsafe-inline'` to manifest CSP |
|
||||
| "No GPU processes" when Immich is processing | Jobs may be between batches | Check `nvidia-smi` directly — should show python process |
|
||||
| Ctrl+Shift+R needed after updates | Cockpit aggressively caches package content | Hard-refresh or open in private/incognito tab |
|
||||
|
||||
## Related
|
||||
|
||||
- Install `nvtop` for terminal TUI monitor: `apt install nvtop` (or build from https://github.com/Syllo/nvtop)
|
||||
- GPU acceleration setup: see the parent `docker-gpu-acceleration` skill
|
||||
@@ -0,0 +1,103 @@
|
||||
# Immich ML GPU Cache Paths and Model Names
|
||||
|
||||
Known working for Immich v2.7.5.
|
||||
|
||||
## Model Sources
|
||||
|
||||
All models hosted on HuggingFace under `immich-app/` org.
|
||||
|
||||
| Model Name | HuggingFace Repo | Files | Size |
|
||||
|---|---|---|---|
|
||||
| `ViT-B-32__openai` | `immich-app/ViT-B-32__openai` | visual/model.onnx (335 MB), textual/model.onnx (254 MB) | ~590 MB total |
|
||||
| `buffalo_l` | `immich-app/buffalo_l` | detection/model.onnx (16 MB), recognition/model.onnx (166 MB) | ~182 MB total |
|
||||
|
||||
## Cache Directory Structure
|
||||
|
||||
Cache root: `/cache` (from `MACHINE_LEARNING_CACHE_FOLDER` env var).
|
||||
|
||||
### CLIP (smart search)
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model task | `SEARCH` → `"clip"` |
|
||||
| Model type | `VISUAL` → `"visual"`, `TEXTUAL` → `"textual"` |
|
||||
| Cache dir | `/cache/clip/ViT-B-32__openai` |
|
||||
| Model dir (visual) | `/cache/clip/ViT-B-32__openai/visual` |
|
||||
| Model path (visual) | `/cache/clip/ViT-B-32__openai/visual/model.onnx` |
|
||||
| Model dir (textual) | `/cache/clip/ViT-B-32__openai/textual` |
|
||||
| Model path (textual) | `/cache/clip/ViT-B-32__openai/textual/model.onnx` |
|
||||
|
||||
### Face detection & recognition
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Model task | `FACIAL_RECOGNITION` → `"facial-recognition"` |
|
||||
| Cache dir | `/cache/facial-recognition/buffalo_l` |
|
||||
| Model dir (detection) | `/cache/facial-recognition/buffalo_l/detection` |
|
||||
| Model path (detection) | `/cache/facial-recognition/buffalo_l/detection/model.onnx` |
|
||||
| Model dir (recognition) | `/cache/facial-recognition/buffalo_l/recognition` |
|
||||
| Model path (recognition) | `/cache/facial-recognition/buffalo_l/recognition/model.onnx` |
|
||||
|
||||
## Code Derivation
|
||||
|
||||
Cache paths follow this pattern:
|
||||
|
||||
```python
|
||||
from immich_ml.config import settings
|
||||
# settings.cache_folder = "/cache"
|
||||
# model_task.value = one of: "clip", "facial-recognition", "ocr"
|
||||
# model_name = e.g. "ViT-B-32__openai" or "buffalo_l"
|
||||
# model_type.value = one of: "visual", "textual", "detection", "recognition"
|
||||
|
||||
cache_dir = settings.cache_folder / model_task.value / model_name
|
||||
model_dir = cache_dir / model_type.value
|
||||
model_path = model_dir / "model.onnx"
|
||||
```
|
||||
|
||||
## Pre-download command
|
||||
|
||||
```python
|
||||
from huggingface_hub import snapshot_download
|
||||
import os
|
||||
|
||||
# For CLIP
|
||||
os.makedirs("/cache/clip/ViT-B-32__openai", exist_ok=True)
|
||||
snapshot_download(
|
||||
"immich-app/ViT-B-32__openai",
|
||||
cache_dir="/cache/clip/ViT-B-32__openai",
|
||||
local_dir="/cache/clip/ViT-B-32__openai",
|
||||
ignore_patterns=["*.armnn", "*.rknn"],
|
||||
)
|
||||
|
||||
# For face detection/recognition
|
||||
os.makedirs("/cache/facial-recognition/buffalo_l", exist_ok=True)
|
||||
snapshot_download(
|
||||
"immich-app/buffalo_l",
|
||||
cache_dir="/cache/facial-recognition/buffalo_l",
|
||||
local_dir="/cache/facial-recognition/buffalo_l",
|
||||
ignore_patterns=["*.armnn", "*.rknn"],
|
||||
)
|
||||
```
|
||||
|
||||
## docker-compose GPU config
|
||||
|
||||
Required addition to `immich-machine-learning` service:
|
||||
|
||||
```yaml
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
## Environment variables
|
||||
|
||||
| Variable | Value | Effect |
|
||||
|---|---|---|
|
||||
| `DEVICE` | `cuda` | Use CUDA (set by -cuda image) |
|
||||
| `MACHINE_LEARNING_MODEL_ARENA` | `false` | Load all models at once. Set `true` to load one at a time (saves VRAM but slower switching) |
|
||||
| `MACHINE_LEARNING_DEVICE_ID` | `0` | GPU device index |
|
||||
| `MACHINE_LEARNING_CACHE_FOLDER` | `/cache` | Where models are stored |
|
||||
Reference in New Issue
Block a user