ray/hermes-config
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
hermes-config/skills/gaming/game-streaming/references/headless-troubleshooting.md
T
ray dab5a4ebc6 initial commit
2026-07-12 10:17:17 -04:00

11 KiB
Raw Permalink Blame History

Headless Sunshine Troubleshooting

Real-world issues and fixes from a headless Sunshine + Moonlight setup on Ubuntu 26.04 with NVIDIA RTX 2080 Ti (driver 580.159.03).

Symptom: "Video failed to find working encoder" / "Unable to find display or encoder"

Sunshine starts but immediately exits. Logs show it trying vulkan → vaapi → software, all failing, never reaching NVENC.

Root cause: cascade of silent failures

Several things can cause this, often in combination:

  1. User not in video and render groups — can't access /dev/dri/card0 or /dev/dri/renderD128
  2. Systemd service missing DISPLAY — X11 capture can't connect to Xorg
  3. CUDA/driver mismatch — NVFBC capture fails, and if X11 fallback is also broken, nothing works

Debugging technique: run sunshine directly

Systemd journal logs are terse — "nvenc failed" with no detail. Run sunshine from the terminal to see the full verbose output:

systemctl --user stop app-dev.lizardbyte.app.Sunshine
DISPLAY=:0 /usr/bin/sunshine ~/.config/sunshine/sunshine.conf

The verbose output shows every CUDA symbol load, every NVENC initialization call, and the exact failure point. In the working case, you'll see:

Found H.264 encoder: h264_nvenc [nvenc]
Found HEVC encoder: hevc_nvenc [nvenc]

Note: When checking DRM device permissions, use stat /dev/dri/card0 — do NOT cat or dd from DRM devices as they can hang the shell when read without a proper DRM master context.

Fix 1: Group permissions

# Check current groups
groups $USER | grep -oP '(video|render|input|audio)'

# Add missing groups
sudo usermod -a -G video,render,input,audio $USER

# Refresh groups in current session (no re-login needed for systemd user)
systemctl --user daemon-reexec

# Fix uinput for gamepad support
sudo chown $USER:input /dev/uinput

Without these:

  • /dev/dri/card0 → "Permission denied" → no KMS capture
  • /dev/dri/renderD128 → can't initialize NVENC CUDA context
  • /dev/uinput → "Gamepad ds5 is disabled due to Permission denied"
  • /dev/snd/* → PipeWire can't enumerate ALSA hardware (auto_null sink still works for capture, but NVIDIA HDMI audio sink won't appear until after reboot)

Fix 1b: Give Sunshine effective cap_sys_admin for input injection

Sunshine has cap_sys_admin=pe (permitted only) by default and drops it on startup (log: drop_elevated_privileges succeeded in dropping capabilities). After dropping, /dev/uinput access may fail silently even when the user is in the input group — no visible error, just zero input events in the log.

# Check current capabilities
getcap /usr/bin/sunshine
# cap_sys_admin,cap_sys_nice=p  ← 'p' = permitted only, dropped on startup

# Set to effective (keeps the capability active at runtime)
sudo setcap cap_sys_admin=ep /usr/bin/sunshine
getcap /usr/bin/sunshine
# cap_sys_admin=ep  ← 'e' = effective, won't be dropped

Without cap_sys_admin=ep, Sunshine processes zero input events from Moonlight — the log shows stream activity but no XTEST, keyboard, or mouse entries. With it, you'll see XTEST relative movement and key events in the log as the client sends input.

Check if uinput is available

On modern kernels (Ubuntu 7.0+), uinput is often built-in rather than a loadable module:

# Check if uinput is built-in or a module
modinfo uinput | head -3
# filename: (builtin)  ← no need to modprobe, already available
# OR: filename: /lib/modules/.../uinput.ko  ← loadable, may need modprobe

# Verify the device node exists with correct permissions
stat /dev/uinput
# crw-rw---- 1 root input 10, 223 ...

# Ray must be in the input group to access it
groups $USER | grep input

Do NOT waste time running modprobe uinput if it's built-in — it will silently succeed without loading anything. The key checks are: device node exists, user in input group, and cap_sys_admin=ep on the binary.

Fix 2: Systemd DISPLAY override

The sunshine systemd service doesn't know about manually-started Xorg sessions. Create a drop-in:

mkdir -p ~/.config/systemd/user/app-dev.lizardbyte.app.Sunshine.service.d
cat > ~/.config/systemd/user/app-dev.lizardbyte.app.Sunshine.service.d/override.conf << 'EOF'
[Service]
Environment=DISPLAY=:0
# Clear the 5-second sleep delay (was waiting for desktop session)
ExecStartPre=
LimitNICE=-10
EOF

systemctl --user daemon-reload
systemctl --user restart app-dev.lizardbyte.app.Sunshine

Without this, X11 capture fails with "Unable to initialize capture method."

Pitfall: Do NOT add SupplementaryGroups=input to this override. If the user's login session doesn't have the input group (because group changes need a fresh login), systemd will fail to start the service with status=216/GROUP — "Changing group credentials failed: Operation not permitted." Adding supplementary groups only works if the user manager itself has those groups. For gamepad support, fix uinput permissions instead (see Fix 1), or wait until next login for group changes to propagate.

Fix 2b: output_name must match the connected display

Sunshine's output_name is a zero-based index into the list of detected displays. If it points to a disconnected output, streaming will produce a black screen or fall back to "recording entire virtual desktop." Check which displays are connected:

# Method 1: xrandr
DISPLAY=:0 xrandr --listmonitors
# Shows: Monitors: 1
#  0: +*HDMI-0 3840/697x2160/392+0+0  HDMI-0
# The monitor name is HDMI-0, but output_name needs the INDEX (0 in this case)

# Method 2: Check sunshine log
grep "Detected display" ~/.config/sunshine/sunshine.log
# Shows: Detected display: DP-0 (id: 0)DP-0 connected: false
#        Detected display: HDMI-0 (id: 2)HDMI-0 connected: true
# → Set output_name = 2

Sunshine logs each display as it detects them with (id: N) and connected: true/false. Match the index to the one that's connected: true. On headless systems with a dummy plug, only one display will be connected. Setting the wrong index produces: Warning: Couldn't get requested display info, defaulting to recording entire virtual desktop.

Fix 3: CUDA/driver version mismatch (NVFBC → X11 fallback)

Sunshine v2026+ bundles CUDA 13.1.1 which requires NVIDIA driver ≥590.48.01. On systems with older drivers (e.g., 580.159.03), NVFBC capture fails because the CUDA runtime can't initialize. However, NVENC encoding works fine — the NVIDIA encode library (libnvidia-encode.so.1) is driver-versioned separately and doesn't need CUDA 13.

Solution: Force X11 capture instead of NVFBC:

# ~/.config/sunshine/sunshine.conf
capture = x11
encoder = nvenc
adapter_name = /dev/dri/renderD128
output_name = 0

X11 capture + NVENC works reliably. NVFBC is faster (direct GPU memory) but requires compatible CUDA. KMS capture also works but needs video group and cap_sys_admin.

Verifying NVENC works independently

Even when Sunshine's NVFBC path fails, ffmpeg can confirm NVENC is functional:

DISPLAY=:0 ffmpeg -f x11grab -framerate 30 -video_size 1920x1080 -i :0.0+0,0 \
  -c:v h264_nvenc -preset p1 -t 3 -f null /dev/null

If this works (shows encoding fps), NVENC is fine — the problem is Sunshine's capture/init path.

Fix 4: CSRF protection for remote web UI access

Sunshine v2026 added CSRF protection. Accessing the web UI from anything other than localhost is blocked. Add to sunshine.conf:

origin_pin_allowed = pc
csrf_allowed_origins = https://100.93.253.36:47990,https://192.168.50.98:47990

Both origin_pin_allowed = pc and csrf_allowed_origins are required for LAN access. origin_pin_allowed relaxes the PIN/credential creation check for LAN origins; csrf_allowed_origins whitelists the specific origin URLs for CSRF token validation. Without origin_pin_allowed, credential creation is blocked even with the CSRF origin whitelisted.

Replace the IPs with your server's LAN and Tailscale IPs. Multiple origins are comma-separated. Restart sunshine after changing.

Fix 5: Start-limit hit from repeated failures

After too many restart attempts, systemd blocks further starts:

app-dev.lizardbyte.app.Sunshine.service: Start request repeated too quickly.
Failed with result 'start-limit-hit'.

Reset with:

systemctl --user reset-failed app-dev.lizardbyte.app.Sunshine
systemctl --user start app-dev.lizardbyte.app.Sunshine

Working sunshine.conf (headless X11 + NVENC)

sunshine_name = rayserver
capture = x11
encoder = nvenc
# adapter_name should NOT be set for NVENC — Sunshine auto-detects the GPU
# adapter_name = auto
output_name = 0
min_log_level = info
upnp = disabled
origin_pin_allowed = pc
csrf_allowed_origins = https://100.93.253.36:47990
audio_sink = auto_null
virtual_sink = auto_null

IMPORTANT: output_name is the zero-based index of the connected display. It varies per system. Check with grep "Detected display" ~/.config/sunshine/sunshine.log after starting Sunshine — use the index of the display marked connected: true. See Fix 2b above for details.

Fix 6: Root process picks NvFBC (no input forwarding)

When Sunshine runs as root (via sudo systemd-run, a system-level service, or direct sudo sunshine), it auto-selects NvFBC capture because root has direct GPU access. NvFBC reads the framebuffer from GPU memory — it bypasses X11 entirely. Result: video streams fine, audio works, but zero input events reach the desktop. Sunshine logs show no XTEST, keyboard, or mouse activity at all.

The fix is simple — don't run as root. Use the user-level systemd service:

# STOP: never do this
sudo systemd-run --unit=sunshine-streamer sunshine   # → NvFBC, no input

# DO: always use the user service
systemctl --user start app-dev.lizardbyte.app.Sunshine

# Verify it's running as the right user
ps aux | grep sunshine | grep -v grep
# Should show: ray ... /usr/bin/sunshine  ← NOT root

If you accidentally started as root, kill it and restart:

sudo pkill -f "sunshine"  # kills all sunshine processes (including root one)
sleep 2
systemctl --user start app-dev.lizardbyte.app.Sunshine

Verification checklist

# 1. Xorg running on dummy display
DISPLAY=:0 xdpyinfo | grep dimensions   # should show 3840x2160

# 2. Groups correct
groups $USER | grep -E 'video|render|input|audio'

# 3. NVENC encoders detected by ffmpeg
ffmpeg -encoders 2>/dev/null | grep nvenc

# 4. Sunshine running
systemctl --user status app-dev.lizardbyte.app.Sunshine

# 5. Web UI accessible
curl -sk https://localhost:47990/ | head -5

# 6. Avahi advertising
avahi-browse -at 2>/dev/null | grep sunshine

# 7. Audio running (no warnings in sunshine logs)
pactl info | grep "Default Sink"
journalctl --user -u app-dev.lizardbyte.app.Sunshine --no-pager -n 30 | grep -i "no audio"  # should return nothing
Reference in New Issue View Git Blame Copy Permalink