hermes-config/skills/self-hosting/docker-gpu-acceleration/references/cockpit-gpu-dashboard.md

# Cockpit NVIDIA GPU Dashboard

> Custom Cockpit package that displays live GPU stats (model, driver, CUDA
> version, temperature, utilization, memory, processes) in the web UI sidebar
> on port 9090.

## When to use

- User wants GPU temperature / utilization / memory visible in Cockpit
- User prefers a web UI over terminal `nvidia-smi` or `nvtop`

## Two approaches

`cockpit.script()` (run nvidia-smi directly) **silently fails on some Cockpit
versions** (Ubuntu 26+, Cockpit ~314+). The **systemd + file-read** approach
is more reliable and preferred.

---

## Approach A (recommended): systemd service + file-read

Write GPU data to files periodically via a background service, then read them
from the Cockpit page with `cockpit.file().read()`. Avoids all PATH and
permission issues with running nvidia-smi from the Cockpit bridge.

### 1. Create the data-collection script

**`/usr/local/bin/nvidia-gpu-monitor.sh`:**

```bash
#!/bin/bash
while true; do
  nvidia-smi --query-gpu=name,driver_version,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw,pstate,fan.speed --format=csv,noheader,nounits 2>/dev/null | head -1 > /run/nvidia-gpu.txt
  nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv,noheader,nounits 2>/dev/null > /run/nvidia-processes.txt
  nvidia-smi 2>/dev/null | grep "CUDA Version" | sed 's/.*CUDA Version: //' | head -1 > /run/nvidia-cuda.txt
  sleep 5
done
```

Make executable: `sudo chmod +x /usr/local/bin/nvidia-gpu-monitor.sh`

### 2. Create systemd service

**`/etc/systemd/system/nvidia-gpu-monitor.service`:**

```ini
[Unit]
Description=NVIDIA GPU Monitor - writes stats to /run
After=nvidia-persistenced.service

[Service]
Type=simple
ExecStart=/usr/local/bin/nvidia-gpu-monitor.sh
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
```

Enable and start:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now nvidia-gpu-monitor
```

### 3. Create the Cockpit package

**`/usr/share/cockpit/nvidia-gpu/manifest.json`:**

```json
{
    "version": 1,
    "requires": { "cockpit": "260" },
    "menu": {
        "index": { "label": "NVIDIA GPU", "order": 90 }
    },
    "content-security-policy": "default-src 'self'; script-src 'unsafe-inline'"
}
```

**`/usr/share/cockpit/nvidia-gpu/index.html`:**

The page uses `cockpit.file(path).read()` to read `/run/nvidia-gpu.txt`,
`/run/nvidia-processes.txt`, and `/run/nvidia-cuda.txt`. Each file contains
a single line of comma-separated values written by the systemd service.

Key implementation:
```javascript
function readFile(path) { return cockpit.file(path).read(); }

function update() {
  readFile('/run/nvidia-gpu.txt').done(function(data) {
    if (!data || !data.trim()) { /* show error */ return; }
    var parts = data.trim().split(', ');
    // parts[0] = name, [1] = driver, [2] = temp, [3] = util,
    // [4] = mem_used, [5] = mem_total, [6] = power, [7] = pstate, [8] = fan
    // ... render cards ...
  });

  readFile('/run/nvidia-cuda.txt').done(function(d) { /* show CUDA version */ });
  readFile('/run/nvidia-processes.txt').done(function(d) { /* render process table */ });
}
setInterval(update, 5000);
```

### 4. Install and restart Cockpit

```bash
sudo mkdir -p /usr/share/cockpit/nvidia-gpu
# Write manifest.json and index.html (use sudo tee or sudo cp)
sudo chmod 644 /usr/share/cockpit/nvidia-gpu/*
sudo systemctl restart cockpit
```

### 5. Verify

1. Open `https://<server>:9090` in browser
2. Log in with server credentials
3. Click **NVIDIA GPU** in sidebar — live metrics refresh every 5s

---

## Approach B (fallback): cockpit.script() directly

Use when Approach A isn't practical (no systemd, containerized Cockpit, etc.):

```javascript
cockpit.script("/usr/bin/nvidia-smi --query-gpu=... --format=csv,noheader,nounits", null, {superuser: 'try'})
  .done(function(data) { /* parse and render */ })
  .fail(function(err) { /* show error */ });
```

**Caveats (why Approach A is preferred):**
- `cockpit.script()` may silently hang on Cockpit v314+ (Ubuntu 26+)
- PATH may not include `/usr/bin/` in non-interactive bridge sessions
- The `.fail()` handler may catch but not always fire for certain errors
- Debugging is harder — the error object from Cockpit bridge is opaque

If using this approach, use the full path `/usr/bin/nvidia-smi` and avoid
`{superuser: 'try'}` unless nvidia-smi requires root on the target system.

---

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| "No NVIDIA GPU detected" | Can't read data files or run nvidia-smi | Check `/run/nvidia-gpu.txt` exists and is populated. Restart service: `sudo systemctl restart nvidia-gpu-monitor` |
| Page shows stale data | systemd service stopped or crashed | `sudo systemctl status nvidia-gpu-monitor` — restart if dead |
| Power shows `[N/A]` | GTX 1050 Ti doesn't report power draw | Normal — handle gracefully in display |
| Page loads blank | CSP blocking inline script | Add `'unsafe-inline'` to manifest CSP |
| "No GPU processes" when Immich is processing | Jobs may be between batches | Check `nvidia-smi` directly — should show python process |
| Ctrl+Shift+R needed after updates | Cockpit aggressively caches package content | Hard-refresh or open in private/incognito tab |

## Related

- Install `nvtop` for terminal TUI monitor: `apt install nvtop` (or build from https://github.com/Syllo/nvtop)
- GPU acceleration setup: see the parent `docker-gpu-acceleration` skill