initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,159 @@
+# Cockpit NVIDIA GPU Dashboard
+
+> Custom Cockpit package that displays live GPU stats (model, driver, CUDA
+> version, temperature, utilization, memory, processes) in the web UI sidebar
+> on port 9090.
+
+## When to use
+
+- User wants GPU temperature / utilization / memory visible in Cockpit
+- User prefers a web UI over terminal `nvidia-smi` or `nvtop`
+
+## Two approaches
+
+`cockpit.script()` (run nvidia-smi directly) **silently fails on some Cockpit
+versions** (Ubuntu 26+, Cockpit ~314+). The **systemd + file-read** approach
+is more reliable and preferred.
+
+---
+
+## Approach A (recommended): systemd service + file-read
+
+Write GPU data to files periodically via a background service, then read them
+from the Cockpit page with `cockpit.file().read()`. Avoids all PATH and
+permission issues with running nvidia-smi from the Cockpit bridge.
+
+### 1. Create the data-collection script
+
+**`/usr/local/bin/nvidia-gpu-monitor.sh`:**
+
+```bash
+#!/bin/bash
+while true; do
+  nvidia-smi --query-gpu=name,driver_version,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw,pstate,fan.speed --format=csv,noheader,nounits 2>/dev/null | head -1 > /run/nvidia-gpu.txt
+  nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits 2>/dev/null > /run/nvidia-processes.txt
+  nvidia-smi 2>/dev/null | grep "CUDA Version" | sed 's/.*CUDA Version: //' | head -1 > /run/nvidia-cuda.txt
+  sleep 5
+done
+```
+
+Make executable: `sudo chmod +x /usr/local/bin/nvidia-gpu-monitor.sh`
+
+### 2. Create systemd service
+
+**`/etc/systemd/system/nvidia-gpu-monitor.service`:**
+
+```ini
+[Unit]
+Description=NVIDIA GPU Monitor - writes stats to /run
+After=nvidia-persistenced.service
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/nvidia-gpu-monitor.sh
+Restart=always
+RestartSec=3
+
+[Install]
+WantedBy=multi-user.target
+```
+
+Enable and start:
+```bash
+sudo systemctl daemon-reload
+sudo systemctl enable --now nvidia-gpu-monitor
+```
+
+### 3. Create the Cockpit package
+
+**`/usr/share/cockpit/nvidia-gpu/manifest.json`:**
+
+```json
+{
+    "version": 1,
+    "requires": { "cockpit": "260" },
+    "menu": {
+        "index": { "label": "NVIDIA GPU", "order": 90 }
+    },
+    "content-security-policy": "default-src 'self'; script-src 'unsafe-inline'"
+}
+```
+
+**`/usr/share/cockpit/nvidia-gpu/index.html`:**
+
+The page uses `cockpit.file(path).read()` to read `/run/nvidia-gpu.txt`,
+`/run/nvidia-processes.txt`, and `/run/nvidia-cuda.txt`. Each file contains
+a single line of comma-separated values written by the systemd service.
+
+Key implementation:
+```javascript
+function readFile(path) { return cockpit.file(path).read(); }
+
+function update() {
+  readFile('/run/nvidia-gpu.txt').done(function(data) {
+    if (!data || !data.trim()) { /* show error */ return; }
+    var parts = data.trim().split(', ');
+    // parts[0] = name, [1] = driver, [2] = temp, [3] = util,
+    // [4] = mem_used, [5] = mem_total, [6] = power, [7] = pstate, [8] = fan
+    // ... render cards ...
+  });
+
+  readFile('/run/nvidia-cuda.txt').done(function(d) { /* show CUDA version */ });
+  readFile('/run/nvidia-processes.txt').done(function(d) { /* render process table */ });
+}
+setInterval(update, 5000);
+```
+
+### 4. Install and restart Cockpit
+
+```bash
+sudo mkdir -p /usr/share/cockpit/nvidia-gpu
+# Write manifest.json and index.html (use sudo tee or sudo cp)
+sudo chmod 644 /usr/share/cockpit/nvidia-gpu/*
+sudo systemctl restart cockpit
+```
+
+### 5. Verify
+
+1. Open `https://<server>:9090` in browser
+2. Log in with server credentials
+3. Click **NVIDIA GPU** in sidebar — live metrics refresh every 5s
+
+---
+
+## Approach B (fallback): cockpit.script() directly
+
+Use when Approach A isn't practical (no systemd, containerized Cockpit, etc.):
+
+```javascript
+cockpit.script("/usr/bin/nvidia-smi --query-gpu=... --format=csv,noheader,nounits", null, {superuser: 'try'})
+  .done(function(data) { /* parse and render */ })
+  .fail(function(err) { /* show error */ });
+```
+
+**Caveats (why Approach A is preferred):**
+- `cockpit.script()` may silently hang on Cockpit v314+ (Ubuntu 26+)
+- PATH may not include `/usr/bin/` in non-interactive bridge sessions
+- The `.fail()` handler may catch but not always fire for certain errors
+- Debugging is harder — the error object from Cockpit bridge is opaque
+
+If using this approach, use the full path `/usr/bin/nvidia-smi` and avoid
+`{superuser: 'try'}` unless nvidia-smi requires root on the target system.
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely Cause | Fix |
+|---------|-------------|-----|
+| "No NVIDIA GPU detected" | Can't read data files or run nvidia-smi | Check `/run/nvidia-gpu.txt` exists and is populated. Restart service: `sudo systemctl restart nvidia-gpu-monitor` |
+| Page shows stale data | systemd service stopped or crashed | `sudo systemctl status nvidia-gpu-monitor` — restart if dead |
+| Power shows `[N/A]` | GTX 1050 Ti doesn't report power draw | Normal — handle gracefully in display |
+| Page loads blank | CSP blocking inline script | Add `'unsafe-inline'` to manifest CSP |
+| "No GPU processes" when Immich is processing | Jobs may be between batches | Check `nvidia-smi` directly — should show python process |
+| Ctrl+Shift+R needed after updates | Cockpit aggressively caches package content | Hard-refresh or open in private/incognito tab |
+
+## Related
+
+- Install `nvtop` for terminal TUI monitor: `apt install nvtop` (or build from https://github.com/Syllo/nvtop)
+- GPU acceleration setup: see the parent `docker-gpu-acceleration` skill