hermes-config/skills/gaming/game-streaming/references/headless-troubleshooting.md

# Headless Sunshine Troubleshooting

Real-world issues and fixes from a headless Sunshine + Moonlight setup on Ubuntu 26.04 with NVIDIA RTX 2080 Ti (driver 580.159.03).

## Symptom: "Video failed to find working encoder" / "Unable to find display or encoder"

Sunshine starts but immediately exits. Logs show it trying vulkan → vaapi → software, all failing, never reaching NVENC.

### Root cause: cascade of silent failures

Several things can cause this, often in combination:

1. **User not in `video` and `render` groups** — can't access `/dev/dri/card0` or `/dev/dri/renderD128`
2. **Systemd service missing `DISPLAY`** — X11 capture can't connect to Xorg
3. **CUDA/driver mismatch** — NVFBC capture fails, and if X11 fallback is also broken, nothing works

### Debugging technique: run sunshine directly

Systemd journal logs are terse — "nvenc failed" with no detail. Run sunshine from the terminal to see the full verbose output:

```bash
systemctl --user stop app-dev.lizardbyte.app.Sunshine
DISPLAY=:0 /usr/bin/sunshine ~/.config/sunshine/sunshine.conf
```

The verbose output shows every CUDA symbol load, every NVENC initialization call, and the exact failure point. In the working case, you'll see:
```
Found H.264 encoder: h264_nvenc [nvenc]
Found HEVC encoder: hevc_nvenc [nvenc]
```

**Note:** When checking DRM device permissions, use `stat /dev/dri/card0` — do NOT `cat` or `dd` from DRM devices as they can hang the shell when read without a proper DRM master context.

## Fix 1: Group permissions

```bash
# Check current groups
groups $USER | grep -oP '(video|render|input|audio)'

# Add missing groups
sudo usermod -a -G video,render,input,audio $USER

# Refresh groups in current session (no re-login needed for systemd user)
systemctl --user daemon-reexec

# Fix uinput for gamepad support
sudo chown $USER:input /dev/uinput
```

Without these:
- `/dev/dri/card0` → "Permission denied" → no KMS capture
- `/dev/dri/renderD128` → can't initialize NVENC CUDA context
- `/dev/uinput` → "Gamepad ds5 is disabled due to Permission denied"
- `/dev/snd/*` → PipeWire can't enumerate ALSA hardware (auto_null sink still works for capture, but NVIDIA HDMI audio sink won't appear until after reboot)

## Fix 1b: Give Sunshine effective `cap_sys_admin` for input injection

Sunshine has `cap_sys_admin=pe` (permitted only) by default and **drops it on startup** (log: `drop_elevated_privileges succeeded in dropping capabilities`). After dropping, `/dev/uinput` access may fail silently even when the user is in the `input` group — no visible error, just zero input events in the log.

```bash
# Check current capabilities
getcap /usr/bin/sunshine
# cap_sys_admin,cap_sys_nice=p  ← 'p' = permitted only, dropped on startup

# Set to effective (keeps the capability active at runtime)
sudo setcap cap_sys_admin=ep /usr/bin/sunshine
getcap /usr/bin/sunshine
# cap_sys_admin=ep  ← 'e' = effective, won't be dropped
```

Without `cap_sys_admin=ep`, Sunshine processes zero input events from Moonlight — the log shows stream activity but no `XTEST`, `keyboard`, or `mouse` entries. With it, you'll see `XTEST relative movement` and key events in the log as the client sends input.

### Check if uinput is available

On modern kernels (Ubuntu 7.0+), `uinput` is often built-in rather than a loadable module:

```bash
# Check if uinput is built-in or a module
modinfo uinput | head -3
# filename: (builtin)  ← no need to modprobe, already available
# OR: filename: /lib/modules/.../uinput.ko  ← loadable, may need modprobe

# Verify the device node exists with correct permissions
stat /dev/uinput
# crw-rw---- 1 root input 10, 223 ...

# Ray must be in the input group to access it
groups $USER | grep input
```

Do NOT waste time running `modprobe uinput` if it's built-in — it will silently succeed without loading anything. The key checks are: device node exists, user in `input` group, and `cap_sys_admin=ep` on the binary.

## Fix 2: Systemd DISPLAY override

The sunshine systemd service doesn't know about manually-started Xorg sessions. Create a drop-in:

```bash
mkdir -p ~/.config/systemd/user/app-dev.lizardbyte.app.Sunshine.service.d
cat > ~/.config/systemd/user/app-dev.lizardbyte.app.Sunshine.service.d/override.conf << 'EOF'
[Service]
Environment=DISPLAY=:0
# Clear the 5-second sleep delay (was waiting for desktop session)
ExecStartPre=
LimitNICE=-10
EOF

systemctl --user daemon-reload
systemctl --user restart app-dev.lizardbyte.app.Sunshine
```

Without this, X11 capture fails with "Unable to initialize capture method."

**Pitfall: Do NOT add `SupplementaryGroups=input` to this override.** If the user's login session doesn't have the `input` group (because group changes need a fresh login), systemd will fail to start the service with `status=216/GROUP` — "Changing group credentials failed: Operation not permitted." Adding supplementary groups only works if the user manager itself has those groups. For gamepad support, fix uinput permissions instead (see Fix 1), or wait until next login for group changes to propagate.

## Fix 2b: `output_name` must match the connected display

Sunshine's `output_name` is a zero-based index into the list of detected displays. If it points to a disconnected output, streaming will produce a black screen or fall back to "recording entire virtual desktop." Check which displays are connected:

```bash
# Method 1: xrandr
DISPLAY=:0 xrandr --listmonitors
# Shows: Monitors: 1
#  0: +*HDMI-0 3840/697x2160/392+0+0  HDMI-0
# The monitor name is HDMI-0, but output_name needs the INDEX (0 in this case)

# Method 2: Check sunshine log
grep "Detected display" ~/.config/sunshine/sunshine.log
# Shows: Detected display: DP-0 (id: 0)DP-0 connected: false
#        Detected display: HDMI-0 (id: 2)HDMI-0 connected: true
# → Set output_name = 2
```

Sunshine logs each display as it detects them with `(id: N)` and `connected: true/false`. Match the index to the one that's `connected: true`. On headless systems with a dummy plug, only one display will be connected. Setting the wrong index produces: `Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop.`

## Fix 3: CUDA/driver version mismatch (NVFBC → X11 fallback)

Sunshine v2026+ bundles CUDA 13.1.1 which requires NVIDIA driver ≥590.48.01. On systems with older drivers (e.g., 580.159.03), NVFBC capture fails because the CUDA runtime can't initialize. However, NVENC encoding works fine — the NVIDIA encode library (`libnvidia-encode.so.1`) is driver-versioned separately and doesn't need CUDA 13.

**Solution:** Force X11 capture instead of NVFBC:

```ini
# ~/.config/sunshine/sunshine.conf
capture = x11
encoder = nvenc
adapter_name = /dev/dri/renderD128
output_name = 0
```

X11 capture + NVENC works reliably. NVFBC is faster (direct GPU memory) but requires compatible CUDA. KMS capture also works but needs `video` group and `cap_sys_admin`.

### Verifying NVENC works independently

Even when Sunshine's NVFBC path fails, ffmpeg can confirm NVENC is functional:

```bash
DISPLAY=:0 ffmpeg -f x11grab -framerate 30 -video_size 1920x1080 -i :0.0+0,0 \
  -c:v h264_nvenc -preset p1 -t 3 -f null /dev/null
```

If this works (shows encoding fps), NVENC is fine — the problem is Sunshine's capture/init path.

## Fix 4: CSRF protection for remote web UI access

Sunshine v2026 added CSRF protection. Accessing the web UI from anything other than localhost is blocked. Add to sunshine.conf:

```ini
origin_pin_allowed = pc
csrf_allowed_origins = https://100.93.253.36:47990,https://192.168.50.98:47990
```

Both `origin_pin_allowed = pc` and `csrf_allowed_origins` are required for LAN access. `origin_pin_allowed` relaxes the PIN/credential creation check for LAN origins; `csrf_allowed_origins` whitelists the specific origin URLs for CSRF token validation. Without `origin_pin_allowed`, credential creation is blocked even with the CSRF origin whitelisted.

Replace the IPs with your server's LAN and Tailscale IPs. Multiple origins are comma-separated. Restart sunshine after changing.

## Fix 5: Start-limit hit from repeated failures

After too many restart attempts, systemd blocks further starts:

```
app-dev.lizardbyte.app.Sunshine.service: Start request repeated too quickly.
Failed with result 'start-limit-hit'.
```

Reset with:

```bash
systemctl --user reset-failed app-dev.lizardbyte.app.Sunshine
systemctl --user start app-dev.lizardbyte.app.Sunshine
```

## Working sunshine.conf (headless X11 + NVENC)

```ini
sunshine_name = rayserver
capture = x11
encoder = nvenc
# adapter_name should NOT be set for NVENC — Sunshine auto-detects the GPU
# adapter_name = auto
output_name = 0
min_log_level = info
upnp = disabled
origin_pin_allowed = pc
csrf_allowed_origins = https://100.93.253.36:47990
audio_sink = auto_null
virtual_sink = auto_null
```

**IMPORTANT:** `output_name` is the zero-based index of the connected display. It varies per system. Check with `grep "Detected display" ~/.config/sunshine/sunshine.log` after starting Sunshine — use the index of the display marked `connected: true`. See Fix 2b above for details.

## Fix 6: Root process picks NvFBC (no input forwarding)

When Sunshine runs as root (via `sudo systemd-run`, a system-level service, or direct `sudo sunshine`), it auto-selects NvFBC capture because root has direct GPU access. NvFBC reads the framebuffer from GPU memory — it bypasses X11 entirely. Result: video streams fine, audio works, but **zero input events reach the desktop**. Sunshine logs show no `XTEST`, `keyboard`, or `mouse` activity at all.

The fix is simple — don't run as root. Use the user-level systemd service:

```bash
# STOP: never do this
sudo systemd-run --unit=sunshine-streamer sunshine   # → NvFBC, no input

# DO: always use the user service
systemctl --user start app-dev.lizardbyte.app.Sunshine

# Verify it's running as the right user
ps aux | grep sunshine | grep -v grep
# Should show: ray ... /usr/bin/sunshine  ← NOT root
```

If you accidentally started as root, kill it and restart:

```bash
sudo pkill -f "sunshine"  # kills all sunshine processes (including root one)
sleep 2
systemctl --user start app-dev.lizardbyte.app.Sunshine
```

## Verification checklist

```bash
# 1. Xorg running on dummy display
DISPLAY=:0 xdpyinfo | grep dimensions   # should show 3840x2160

# 2. Groups correct
groups $USER | grep -E 'video|render|input|audio'

# 3. NVENC encoders detected by ffmpeg
ffmpeg -encoders 2>/dev/null | grep nvenc

# 4. Sunshine running
systemctl --user status app-dev.lizardbyte.app.Sunshine

# 5. Web UI accessible
curl -sk https://localhost:47990/ | head -5

# 6. Avahi advertising
avahi-browse -at 2>/dev/null | grep sunshine

# 7. Audio running (no warnings in sunshine logs)
pactl info | grep "Default Sink"
journalctl --user -u app-dev.lizardbyte.app.Sunshine --no-pager -n 30 | grep -i "no audio"  # should return nothing
```