ray/hermes-config
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
ray
2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/server-health-check/references/cockpit-offline-packagekit-fix.md
+73
View File
@@ -0,0 +1,73 @@
# Cockpit: "Cannot refresh cache whilst offline" / PackageKit + NetworkManager Fix
## Symptom
Cockpit's **Updates** page shows:
> Loading available updates failed — Cannot refresh cache whilst offline
Yet the system has full internet connectivity — `ping`, `curl`, and `apt update` all work.
## Root Cause
On Ubuntu Server, **`systemd-networkd`** handles networking, but **PackageKit** (Cockpit's backend for the updates page) checks **NetworkManager's** D-Bus state to determine if the system is online.
When NM is installed but has **no connection profile** configured (because systemd-networkd is doing the actual networking), NM reports:
```
STATE: disconnected CONNECTIVITY: none
```
PackageKit sees this and refuses to refresh the cache — even though the interface is up and routable via systemd-networkd.
## Diagnosis
```bash
# NM says disconnected even though networking works
nmcli general status
# → STATE: disconnected CONNECTIVITY: none
# But systemd-networkd is fine
networkctl status
# → State: routable
# → Online state: partial
# PackageKit confirms it thinks we're offline
busctl get-property org.freedesktop.PackageKit /org/freedesktop/PackageKit org.freedesktop.PackageKit NetworkState
# 0 = Offline, 1 = Limited, 2 = Online
```
## Fix
Since systemd-networkd is doing all the networking, NetworkManager is redundant. Stop and mask it:
```bash
sudo systemctl mask NetworkManager --now
```
Then restart PackageKit so it re-checks the network state without NM:
```bash
sudo systemctl restart packagekit
sleep 2
busctl get-property org.freedesktop.PackageKit /org/freedesktop/PackageKit org.freedesktop.PackageKit NetworkState
# → u 2 (Online)
```
After this, reload Cockpit's updates page — it should work.
## Verification
```bash
# PackageKit should now report Online
busctl get-property org.freedesktop.PackageKit /org/freedesktop/PackageKit org.freedesktop.PackageKit NetworkState
# Expected: u 2
# Confirm networking still works
ping -c 1 8.8.8.8
curl -s --max-time 5 https://google.com -o /dev/null -w "%{http_code}"
```
## Why This Happens
Ubuntu Server 24+ ships with **both** `systemd-networkd` and `NetworkManager` installed. The server installer configures networkd, not NM. But NM's service is still present and starts, sees no connections, and reports "offline." PackageKit only knows how to check NM — it has no fallback for systems that don't use NM.
Masking NM is safe and standard on systems where networkd handles all interfaces.
skills/devops/server-health-check/references/docker-bridge-gateway-ip-loss.md
+116
View File
@@ -0,0 +1,116 @@
# Docker User-Defined Bridge: Lost Gateway IPv4 Address
## Symptom
A Docker container is running, `docker ps` shows it as healthy, and `docker exec` can reach it from inside. But accessing the published port from the **host** or **other machines on the network** hangs/times out.
```
$ curl -v http://localhost:2283/
* Trying 127.0.0.1:2283...
* Connected to localhost (127.0.0.1) port 2283
> GET / HTTP/1.1
> ...
* Request completely sent off
* Operation timed out after 10001 milliseconds with 0 bytes received
```
Docker-proxy is running and listening on the port, but responses never come back.
## Diagnosis
```bash
# docker-proxy is running and forwarding correctly
ss -tlnp | grep <port>
# → LISTEN 0.0.0.0:<port>
# Docker NAT rule exists
sudo iptables -t nat -L DOCKER -n | grep <port>
# → DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:<port> to:172.XX.0.X:<port>
# Docker filter allows traffic
sudo iptables -L DOCKER -n | grep <container_ip>
# But the host can't reach the container's IP
ping -c 2 <container_ip>
# → 100% packet loss
# The bridge interface has NO IPv4 address
ip addr show br-<hash>
# Only shows inet6 fe80::... — NO inet 172.XX.0.1/16
```
## Root Cause
Docker's user-defined bridge networks (created by `docker compose`) need a **gateway IP** on the host-side bridge interface (`br-<hash>`). This IP is normally assigned when the network is first created.
If the bridge loses its IPv4 (e.g. after `docker compose restart` without a full `down`+`up`, or due to a Docker daemon restart), the host has **no route** to the container's subnet:
```bash
ip route | grep 172.18
# → (nothing — no route)
```
docker-proxy still listens and accepts connections, but it forwards to the container's IP via the bridge. Without the host having an IP on that bridge, the forwarded packets can't reach the container and responses can't come back.
## Fix
### Preferred: Full Network Recreate
```bash
cd /opt/<project>/
docker compose down
docker compose up -d
```
This recreates the network fresh, which assigns the gateway IP correctly. **All containers are recreated**, so the network starts clean.
### Quick Patch (if downtime is unacceptable)
```bash
sudo ip addr add <gateway_ip>/<mask> dev br-<hash>
```
Find the gateway IP from `docker network inspect`:
```bash
docker network inspect <name> | grep -A 5 '"Config"'
# "Gateway": "172.18.0.1"
```
Then:
```bash
sudo ip addr add 172.18.0.1/16 dev br-<hash>
```
This restores connectivity immediately but is **ephemeral** — it won't survive a reboot or Docker daemon restart. Use the full `down && up -d` for a permanent fix.
### Making the Quick Patch Persistent
If you need a stopgap before the full restart, create a oneshot systemd service:
```ini
[Unit]
Description=Assign IP to Docker bridge
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
ExecStartPre=/bin/bash -c "sleep 5"
ExecStart=/sbin/ip addr add <gateway_ip>/<mask> dev br-<hash> 2>/dev/null || /bin/true
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
```
```bash
sudo systemctl enable --now <service>
```
## Prevention
- **Avoid `docker compose restart <service>`** for critical containers on user-defined bridge networks if you suspect the network might be degraded
- Use **full `docker compose down && docker compose up -d`** when something feels off with container networking
- After a **full machine reboot**, Docker recreates everything properly — this issue typically only appears during hot-fixes or partial restarts
skills/devops/server-health-check/references/pihole-v6-diagnostics.md
+190
View File
@@ -0,0 +1,190 @@
# Pi-hole v6 Diagnostics
Quick-start for diagnosing "is Pi-hole causing my network issues?" questions. Pi-hole v6 changed the API, CLI, and database schema — the old `pihole -c` (chronometer), `/admin/api.php`, and v5 commands don't work.
## Decision Flow
1. **Check if Pi-hole is even the problem** before digging in. If DNS resolves through Pi-hole and no essential domains are blocked, the issue is at the WiFi/router/IP level — not Pi-hole.
2. If Pi-hole might be the culprit, query its SQLite database directly.
## Docker Status
```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep pihole
```
Pi-hole v6 container info:
- Status should show `healthy`
- Binds port 53 TCP+UDP to host IP (typically `192.168.50.X:53`)
- Web UI on port 80 inside container, mapped to host (e.g., `8090:80`)
## Pi-hole Process Check
```bash
docker exec pihole pihole status
```
Expected output:
```
[✓] FTL is listening on port 53
[✓] UDP (IPv4)
[✓] TCP (IPv4)
[✓] UDP (IPv6)
[✓] TCP (IPv6)
[✓] Pi-hole blocking is enabled
```
## Querying the Database (When Web UI Is Locked)
Pi-hole v6's web API at `/api` requires authentication (session or app password). When locked out, query the SQLite database directly.
### Database Location
- **Inside container:** `/etc/pihole/pihole-FTL.db`
- **On host (Docker volume):** `/var/lib/docker/volumes/pihole_etc/_data/pihole-FTL.db`
### Access Pattern
The `/var/lib/docker` directory has `drwx--x---` permissions — regular users can't traverse it. Copy the DB to a readable location:
```bash
sudo cp /var/lib/docker/volumes/pihole_etc/_data/pihole-FTL.db /tmp/pihole-FTL.db
sudo cp /var/lib/docker/volumes/pihole_etc/_data/pihole-FTL.db-wal /tmp/pihole-FTL.db-wal
sudo cp /var/lib/docker/volumes/pihole_etc/_data/pihole-FTL.db-shm /tmp/pihole-FTL.db-shm
sudo chown $USER:$USER /tmp/pihole-FTL.db*
```
Then query with Python (the Pi-hole container has no Python/sqlite3):
```python
import sqlite3
conn = sqlite3.connect('/tmp/pihole-FTL.db')
cur = conn.cursor()
# Recent queries
cur.execute("""
SELECT datetime(timestamp, 'unixepoch', 'localtime'), domain, client, status
FROM queries ORDER BY id DESC LIMIT 30
""")
```
## Pi-hole v6 Status Codes
Status codes in the `queries` table:
| Code | Meaning | Notes |
|------|---------|-------|
| 1 | GRAVITY_BLOCKED | Domain in blocklist |
| 2 | FORWARDED | Sent to upstream DNS |
| 3 | CACHE | Answered from cache |
| 4 | REGEX_BLOCKED | Matched regex block rule |
| 5 | EXACT_BLOCKED | Exact-match block |
| 6 | UPSTREAM_BLOCKED | Blocked by upstream DNS |
| 9 | CACHE_STALE | Stale cache entry |
| 12 | ALREADY_FORWARDED | Duplicate suppressed |
| 14 | CACHED_BLOCKED | Cached as blocked (from prior CNAME-chain inspection) — **this is the dangerous one for false positives** |
| 16 | ALREADY_BLOCKED (variant) | Gravity already known |
| 17 | RETRIED | First attempt failed, retry succeeded |
**Status 17 is normal** — it's the dominant status in v6 and means the query succeeded on retry. It does NOT indicate a problem.
**Only status 1, 4, 5, 6 are actual blocks.** Everything else = allowed.
## Key Diagnostic Queries
### Blocked vs Allowed Ratio (last hour)
```sql
SELECT CASE WHEN status IN (1,4,5,6) THEN 'BLOCKED' ELSE 'ALLOWED' END, count(*)
FROM queries WHERE timestamp > strftime('%s','now','-1 hour')
GROUP BY 1
```
### Which Clients Are Using Pi-hole
```sql
SELECT client, count(*) FROM queries
WHERE timestamp > strftime('%s','now','-1 hour')
GROUP BY client ORDER BY count(*) DESC
```
### Recent Blocked Domains
```sql
SELECT domain, count(*) FROM queries
WHERE timestamp > strftime('%s','now','-1 hour') AND status IN (1,4,5)
GROUP BY domain ORDER BY count(*) DESC LIMIT 10
```
### Windows NCSI Pattern (Connectivity Check)
Windows devices query `dns.msftncsi.com` every ~60 seconds to verify internet connectivity. If this shows status 17 (RETRIED), the device may be reporting "no internet" even though DNS eventually resolves. This is typically an upstream DNS slowness issue, not a Pi-hole problem.
## 🔥 Primary Pitfall: Deep CNAME Inspection False Positives
**This is the #1 cause of "some devices have no internet after DNS change to Pi-hole."**
When `CNAMEdeepInspect = true` (default in v6), Pi-hole follows CNAME chains. If ANY domain in the chain matches a blocklist entry, the ENTIRE chain is cached as blocked (status 14). The blocklist (e.g., StevenBlack) includes subdomains like `pagead.l.google.com` — these are ad-specific, but CNAME inspection propagates the block to the PARENT `l.google.com`, which kills ALL services on Google infrastructure.
**Affected domains (false positives):**
- `connectivitycheck.gstatic.com` — Android TV/Shield connectivity check
- `dns.msftncsi.com` — Windows NCSI connectivity check
- `google.com` — basic connectivity
- `ota.nvidia.com` — Shield TV updates
- `clients3.google.com`, `android.apis.google.com` — Google Play Services
**Detection:** Look for status 14 on these domains:
```sql
SELECT domain, count(*) FROM queries WHERE status=14
GROUP BY domain ORDER BY count(*) DESC LIMIT 20;
```
**Fix — whitelist critical domains (preferred):**
```bash
docker exec pihole pihole -w connectivitycheck.gstatic.com
docker exec pihole pihole -w dns.msftncsi.com
docker exec pihole pihole -w clients3.google.com
docker exec pihole pihole -w ota.nvidia.com
```
**Alternative — disable deep CNAME inspection:**
Set `CNAMEdeepInspect = false` in `/etc/pihole/pihole.toml` (may allow some CNAME-cloaked ads through).
See also: `pihole-troubleshooting` skill for full diagnostic workflow.
## Red Flags (Pi-hole IS the Problem)
- Essential domains (google.com, microsoft.com, apple.com) appearing with status 1, 4, or 5
- Status 14 on connectivity-check domains (connectivitycheck.gstatic.com, dns.msftncsi.com) — indicates deep CNAME inspection false positive
- Large block percentage (>50%) in last hour
- DNS resolution fails from other hosts: `dig @192.168.50.X google.com` times out
- Pi-hole container unhealthy or in restart loop
## Red Herrings (Pi-hole is NOT the Problem)
- Tracking/analytics domains blocked (app-measurement.com, newrelic.com, doubleclick.net) — these are working as intended
- Status 17 (RETRIED) on queries — normal v6 behavior, queries succeed
- Some devices working while others don't — if Pi-hole DNS resolves for any device, the DNS layer is fine; connectivity issues are at WiFi/router/IP level (but FIRST rule out deep CNAME inspection false positives above)
## Router Investigation (When Pi-hole Is Ruled Out)
Check the ASUS router at 192.168.50.1:
1. **System Log → Wireless Log** — are affected devices connected?
2. **Network Map → Clients** — do they have valid IPs?
3. **WiFi Settings** — Smart Connect / band steering can confuse some devices. Try separating 2.4 GHz and 5 GHz SSIDs.
## Pi-hole v6 API (When You Have the Password)
```bash
# Get session
curl -s -X POST "http://192.168.50.X:8090/api/auth" \
-H "Content-Type: application/json" \
-d '{"password":"YOUR_PASSWORD"}'
# Get summary (after auth)
curl -s "http://192.168.50.X:8090/api/stats/summary" \
-H "Authorization: Bearer SESSION_TOKEN"
```
## Other v6 Changes from v5
- `pihole -c` (chronometer) → removed, use PADD
- `/admin/api.php` → `/api` (different path, authentication required)
- `gravity.db` → replaced by `gravity` table in `pihole-FTL.db`
- Container has no Python or sqlite3 — query from host
skills/devops/server-health-check/references/usb-enclosures-linux.md
+66
View File
@@ -0,0 +1,66 @@
# USB Drive Enclosures & Docks — Linux Compatibility Guide
## Chipset Rankings for Linux
| Chipset | UASP | SMART Passthrough | Reliability | Verdict |
|---------|:----:|:-----------------:|:-----------:|:--------|
| **ASMedia ASM1153E / ASM235CM** | ✅ Excellent | ✅ `-d sat` | ✅ Rock-solid | **Gold standard** |
| **ASMedia ASM1351** | ✅ Good | ✅ `-d sat` | ✅ Good | Found in TerraMaster, QNAP enclosures |
| **JMicron JMS578 / JMS561** | ⚠️ Good | ✅ `-d sat,12` | ⚠️ Occasional UAS abort storms | Acceptable with quirks |
| **Realtek RTL9210B-CG** | ⚠️ Mixed | ❌ Poor for SATA | ⚠️ Intermittent disconnects | **Avoid for HDDs** |
| **VIA VL812 / VL822** | ⚠️ Fair | ❌ Often fails | ⚠️ Inconsistent | Not recommended |
### Key insight
**ASMedia is the only chipset that "just works"** on modern Linux kernels with full UASP + SMART passthrough. JMicron works but can produce the same `uas_eh_abort_handler` errors seen with failing USB drives — making it hard to distinguish a bad chipset from a bad drive. Realtek's SATA mode is unreliable.
## Recommended Models
### Single-bay (for one IronWolf Pro / single backup drive)
| Model | Chipset | Price | Notes |
|-------|---------|-------|-------|
| **Sabrent DS-UB3C1** (dock) | ASMedia ASM1153E | ~$18 | Most popular dock on Linux |
| **Sabrent EC-UASP** (enclosure) | ASMedia ASM1153E | ~$20 | Well-tested enclosure |
| **UGREEN CM121** (enclosure) | ASMedia ASM235CM | ~$24 | Newer chip, runs cool |
| **Startech SATDOCKU3SEF** (dock) | ASMedia ASM1153E | ~$32 | Heavy-duty build |
### Two-bay (JBOD — drives appear as separate devices)
| Model | Chipset | Price | Notes |
|-------|---------|-------|-------|
| **TerraMaster D2-320** | ASMedia ASM235CM + JMB575 | ~$75 | **Best choice** — full UASP + SMART, hardware JBOD dip switch, no kernel quirks |
| **Yottamaster D35-2C** | Realtek RTL9210B-CG | ~$65 | Cheaper, but Realtek bridge can have AMD XHCI disconnect issues |
| **Sabrent DS-2BCR** | ASMedia ASM225CM | ~$100 | Premium build, tool-free trays, silent fan |
| **Mediasonic ProBox HF2-SU3S2** | JMS539 (BOT only, no UASP!) | ~$60 | **Avoid** — no UASP, unreliable SMART |
| **ORICO 2-bay** | VIA VL812 lottery | ~$45 | **Avoid** — chipset lottery, underpowered 24W PSU |
## Power Supply Notes
- **Two 3.5" HDDs peak at ~25W each during spin-up** (12V × ~2A)
- Minimum safe PSU for 2-bay: **12V/3A (36W)** — adequate but marginal
- Recommended: **12V/5A (60W)** brick (~$15 upgrade) for headroom
- ORICO's 12V/2A (24W) PSU is dangerously underpowered for two HDDs
## SMART Verification
After connecting, verify everything works:
```bash
# Confirm UASP driver loaded
lsusb -t | grep uas
# Full SMART data
sudo smartctl -a -d sat /dev/sdX
# Check temperature and power-on hours
sudo smartctl -a -d sat /dev/sdX | grep -E 'Temperature|Power_On_Hours'
```
If `smartctl` returns no data or errors, the enclosure chipset doesn't support SMART passthrough.
## JBOD Mode vs RAID
- **JBOD** (Just a Bunch Of Disks) = each drive appears as a separate `/dev/sdX` — what you want for "backup + backup of backup"
- **RAID 0** = striping (fast, no redundancy)
- **RAID 1** = mirroring (redundant but both drives show as one device — NOT what you want for independent backups)
- **RAID mode on the enclosure does NOT replace software backup** — use it in JBOD mode and let rsync/rclone handle the actual backup logic