initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,284 @@
+---
+name: storage-management
+description: Detect, partition, format, mount, and back up storage drives on Linux servers — including fstab setup, rsync backup workflows, and permission management.
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [storage, disks, drives, rsync, backup, fstab, mount]
+related_skills: [server-health-check]
+---
+
+# Storage & Drive Management
+
+Manage physical storage drives on Linux — detection, partitioning, formatting, persistent mounting, and full-drive backup via rsync.
+
+## Detecting New Drives
+
+```bash
+# Full overview: include TRAN(transport) to distinguish USB from SATA/NVMe
+lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN
+
+# Unmounted drives (device listed but no mountpoint)
+lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN | grep -E 'disk$|part$' | awk '$5 == ""'
+
+# Get UUIDs for fstab
+sudo blkid
+
+# Find by USB vendor/model (useful for identifying thumb drives)
+ls -la /dev/disk/by-id/usb* 2>/dev/null
+```
+
+The `TRAN` column shows `usb`, `sata`, or `nvme` — immediately tells you how the drive is connected.
+A drive showing in `lsblk` with no partitions and no mount point is ready to be set up.
+
+## Partitioning & Formatting
+
+### Step 1 — Create GPT partition table + single partition
+
+```bash
+sudo parted /dev/sdX mklabel gpt
+sudo parted /dev/sdX mkpart primary ext4 0% 100%
+sudo partprobe /dev/sdX    # reload partition table
+```
+
+### Step 2 — Format with ext4 (label it for easy identification)
+
+```bash
+sudo mkfs.ext4 /dev/sdX1 -L DriveLabel
+```
+
+**Note:** `mkfs` is on the unconditional blocklist in Hermes. The agent cannot run it. Ask the user to run the command directly, then proceed with mounting and fstab.
+
+### Step 3 — Create mount point and mount
+
+```bash
+sudo mkdir -p /mnt/drive-label
+sudo mount /dev/sdX1 /mnt/drive-label
+```
+
+### Step 4 — Add to fstab for persistence
+
+```bash
+# Get UUID
+sudo blkid /dev/sdX1
+
+# Add entry (ext4 example)
+echo 'UUID=xxxx-xxxx /mnt/drive-label ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab
+```
+
+**fstab format:** `<file_system> <mount_point> <type> <options> <dump> <pass>`
+
+| Field | Meaning |
+|-------|---------|
+| `UUID=...` | Use UUID, not `/dev/sdX` — survives re-enumeration |
+| mount point | Directory to mount at |
+| type | `ext4`, `ntfs-3g`, `exfat`, `vfat` |
+| options | `defaults`, `noatime` (skip access time updates), `uid=1000,gid=1000,umask=000` (permissions for NTFS/exFAT) |
+| dump | 0 (no backup) |
+| pass | `2` (non-root fsck), `1` (root fsck), `0` (no fsck) |
+
+### External/Removable Drives
+
+**ALWAYS add `nofail` to external drives in fstab.** Without it, systemd waits for the drive at boot and either hangs or drops to emergency mode if the drive is unplugged.
+
+```bash
+# WRONG — hangs at boot if drive missing
+UUID=xxxx /mnt/external ext4 defaults,noatime 0 2
+
+# RIGHT — skips silently if drive not present
+UUID=xxxx /mnt/external ext4 defaults,noatime,nofail 0 0
+```
+
+Also use `pass=0` (last column) for external drives — a missing drive that's flagged for fsck (`0 2`) will cause a boot stall even with `nofail`.
+
+### Step 5 — Verify
+
+```bash
+sudo mount -a    # re-read fstab, mount any missing entries
+df -h /mnt/drive-label
+lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
+```
+
+## Unmounting Drives
+
+Always unmount before physically disconnecting a drive to avoid data corruption.
+
+```bash
+sudo umount /mnt/drive-label
+```
+
+### When umount hangs or times out
+
+If `umount` hangs (e.g., a process still has open file handles or I/O is stalled), use lazy unmount:
+
+```bash
+sudo umount -l /mnt/drive-label
+```
+
+`-l` (lazy) detaches the filesystem from the mount tree immediately and cleans up references once they're no longer in use. The drive is safe to disconnect after `umount -l` returns successfully — you'll either get a clean detach or `not mounted` (meaning the original umount actually completed despite the timeout).
+
+If even `umount -l` reports `not mounted`, the drive is already detached — safe to remove.
+
+### Filesystem-Specific Notes
+
+| FS | Install Package | Mount Type | Special Options |
+|----|----------------|------------|-----------------|
+| **ext4** | built-in | default | `defaults,noatime` |
+| **NTFS** | `ntfs-3g` | `ntfs-3g` | `uid=1000,gid=1000,umask=000` (full user access) |
+| **exFAT** | `exfatprogs` | `exfat` | `uid=1000,gid=1000,umask=000` |
+
+> ⚠️ **NTFS on Linux:** Always specify `-t ntfs-3g` explicitly. The kernel `ntfs3` driver (in-tree) is experimental and can corrupt data. Use the userspace `ntfs-3g` which is battle-tested.
+
+## Full-Drive Backup with rsync
+
+For copying an entire drive (or large directories) to another local drive:
+
+### Single-Instance rsync (Recommended for USB Drives)
+
+Multiple parallel rsync instances on the same USB source cause I/O contention errors (`error in socket IO`, `Broken pipe`, `errors selecting input/output files`). Always use a **single instance** when the source is a USB/external drive.
+
+```bash
+# Local disk-to-disk copy (fastest approach)
+sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-folder/ \
+  --exclude='$RECYCLE.BIN' --exclude='System Volume Information'
+```
+
+| Flag | Meaning |
+|------|---------|
+| `-a` | Archive mode (preserves permissions, timestamps, symlinks) |
+| `-v` | Verbose |
+| `-h` | Human-readable sizes |
+| `-W` | **Whole-file** (skip delta checksum) — faster for local copies |
+| `--progress` | Show per-file transfer progress |
+
+**Why `-W` for local copies:** rsync normally reads blocks of each file to compute checksums for delta transfer. On a local copy there's no network benefit — `-W` just copies the whole file, eliminating the checksum overhead.
+
+For deep rsync best practices (USB I/O contention, ionice scheduling, progress monitoring, post-backup reorganization, performance expectations), see `references/rsync-best-practices.md` (absorbed from `data-migration`).
+
+### Permission Handling
+
+- **Before running non-sudo rsync, check destination ownership:** `ls -la /mnt/dest/` — if the parent directory is root-owned (e.g., `drwxr-xr-x root root`), rsync will fail with `Permission denied` on `mkdir`. Fix: `sudo chown user:user /mnt/dest/` first.
+- If rsync runs with `sudo`, dest dirs are owned by root
+- Subsequent non-sudo rsync runs fail with `Permission denied` on mkdir
+- **Fix:** `sudo chown -R user:user /mnt/dest/` or chmod on target dirs
+- Or just always run the rsync with `sudo`
+
+### Progress Monitoring
+
+```bash
+# Check size transferred so far
+df -h /mnt/dest-drive
+
+# Check process is alive
+ps aux | grep rsync | grep -v grep
+
+# Recent log lines
+tail -5 /path/to/rsync.log
+
+# File count (slow for large dirs)
+find /mnt/dest/backup-folder/ -type f | wc -l
+```
+
+### Expected Speeds
+
+| Source Type | Typical Speed | Notes |
+|-------------|--------------|-------|
+| USB 3.0 HDD → SATA SSD | 80-120 MB/s | Source-limited |
+| USB 2.0 HDD → any | 20-40 MB/s | Interface bottleneck |
+| NVMe → NVMe | 500+ MB/s | Only if both are internal |
+
+### Resume Behavior
+
+rsync automatically resumes partial transfers. Files with matching name, size, and mtime are skipped. You can safely kill and restart — only new/changed files transfer.
+
+## Exposing Drives Over the Network (Samba)
+
+After mounting a drive, you can share it over the network via Samba (SMB/CIFS) so Windows, macOS, and Linux machines on the LAN can access it as a network drive.
+
+### Check Samba Status
+
+```bash
+which smbd && smbd --version   # check if installed
+cat /etc/samba/smb.conf        # existing config
+```
+
+### Add a New Share
+
+Samba uses a simple INI-style config at `/etc/samba/smb.conf`. Append a new share section:
+
+```bash
+sudo tee -a /etc/samba/smb.conf << 'EOF'
+
+[share-name]
+path = /mnt/drive
+browseable = yes
+read only = no
+guest ok = yes
+force user = ray
+create mask = 0777
+directory mask = 0777
+EOF
+```
+
+| Option | Meaning |
+|--------|---------|
+| `guest ok = yes` | No password required on connect |
+| `force user = ray` | All files written over SMB owned by `ray` |
+| `create mask = 0777` | New files get full permissions |
+| `directory mask = 0777` | New directories get full permissions |
+
+### Restart & Verify
+
+```bash
+sudo systemctl restart smbd
+
+# Verify the share shows up
+smbclient -L //localhost -U ray --no-pass | grep share-name
+
+# Find server LAN IP
+ip -4 addr show | grep -oP 'inet \K[^/]+' | grep -v 127.0.0.1
+```
+
+### Connecting from Clients
+
+| OS | How |
+|----|-----|
+| **Windows** | `\\192.168.x.x\share-name` in File Explorer |
+| **macOS** | ⌘K → `smb://192.168.x.x/share-name` |
+| **Linux** | `smb://192.168.x.x/share-name` in file manager |
+
+### Password-Protected Shares (Alternative)
+
+```bash
+sudo smbpasswd -a ray                    # set SMB password (interactive)
+# or from a script:
+echo 'password' | sudo smbpasswd -a -s ray
+```
+
+Then use `valid users = ray` and `guest ok = no` in the share section.
+
+### Pitfalls
+
+- **SMB password ≠ system password.** Setting a Samba password with `smbpasswd` is a separate step from the user's Linux login password.
+- **`$` in share paths:** The `$RECYCLE.BIN` directory on NTFS drives contains a literal `$`. When writing Samba configs or rsync excludes, the `$` must be quoted or escaped to prevent shell variable expansion.
+- **Firewall:** If clients can't connect, check Samba ports: `sudo ufw allow samba` or `sudo ufw allow 139,445/tcp`.
+
+## Pitfalls
+
+- **Mounted drives are root-owned by default** — `cp`, `rsync`, `mkdir`, and any write to a freshly mounted drive will fail with `Permission denied` unless you use `sudo`. After sudo operations, subdirectories are owned by root and non-sudo writes keep failing. Either always use `sudo` for writes to mounted drives, or `sudo chown user:user /mnt/mountpoint/` after mounting to grant user-level access.
+- **fstab doesn't evaluate shell commands** — `UUID=$(sudo blkid ...)` doesn't work in fstab. Use the literal UUID from `sudo blkid`.
+- **External drives MUST have `nofail`** — missing external drives without `nofail` cause systemd to hang at boot (waits 90s per drive) or drop to emergency mode. Always add `nofail` and `pass=0` to fstab entries for removable/external drives.
+- **Parallel rsync on USB drives** causes `I/O error` / `Broken pipe` / `socket IO error`. The USB controller can't handle concurrent reads. Use a single instance.
+- **Drive re-enumeration** — `/dev/sdb` may become `/dev/sdc` after reboot or replug. Always use UUIDs in fstab.
+- **Permissions gap after sudo rsync** — subsequent non-sudo runs fail on directories the first run created as root. Either always use sudo, or chown after the first run.
+- **NTFS write support** — `ntfs-3g` not `ntfs3`. The kernel ntfs3 driver is experimental. Install `ntfs-3g` explicitly.
+- **exFAT on modern Ubuntu** — package is `exfatprogs` (not `exfat-utils` which is deprecated). Ubuntu 24.04+ has kernel exFAT support, but still needs the tools package.
+
+## References
+
+- `references/drive-selection-guide.md` — SMR vs CMR guidance, speed tiers, physical/electrical compatibility checklist, OEM RAM upgrade assessment (for when the user asks "will this drive fit?" or "how does this compare to my current drive?")
+- `references/rsync-best-practices.md` — rsync pattern reference for USB I/O
+- `references/rayserver-shares.md` — rayserver-specific Samba share inventory (absorbed from `samba-nas`)
@@ -0,0 +1,143 @@
+# Drive Selection Guide for Self-Hosted Linux Servers
+
+When the user asks "will this drive fit my setup?" or "how does this compare to my current drive?", use this guide to assess compatibility, performance tier, and real-world impact.
+
+## Compatibility Assessment Checklist
+
+### 1. Physical Form Factor
+
+| Factor | How to Check |
+|--------|-------------|
+| 3.5" vs 2.5" | `lsblk -o NAME,SIZE,MODEL,TRAN` to see current drives |
+| Available bays | Inspect case — or count `ls /sys/class/ata_link/` vs active drives |
+| Full bay? Can swap | If bay count is full, ask user if they want to replace an existing drive |
+
+### 2. Interface (SATA)
+
+Check `ls /sys/class/ata_link/` — each link is a SATA port. Compare against `lsblk -o NAME,TRAN` to find free ports.
+
+```bash
+# Quick free-port check
+ls /sys/class/ata_link/
+# Shows link1, link2, link3, etc.
+
+# Map which links are used: which /dev/sdX maps to which ata port
+ls -la /dev/disk/by-path/ | grep ata
+```
+
+- A free link = free SATA port ✅
+- Need more ports? Add a PCIe SATA card
+
+### 3. Power
+
+Enterprise HDDs draw 6-8W active vs consumer 4-5W. Standard SATA power connector. Desktop PSUs handle this fine unless adding 6+ drives.
+
+### 4. Boot / Controller Compatibility
+
+- Standard SATA AHCI/RAID: any SATA drive works
+- NVMe: check for free M.2 slot (`lspci | grep Non-Volatile` or `lsblk -d -o NAME,TRAN | grep nvme`)
+- HP OEM boards (like HP 8703) have no firmware-level drive compatibility restrictions — any standard SATA or NVMe drive works
+
+## SMR vs CMR — The Key Performance Distinction
+
+| | SMR (Shingled) | CMR (Conventional) |
+|---|---|---|
+| Write mechanism | Tracks overlap like roof shingles — rewriting one requires rewriting a whole band | Tracks are independent — writes go where told |
+| Sequential write (cached) | ~140 MB/s | ~200 MB/s |
+| Sustained write (cache exhausted) | **30-50 MB/s** — massive dropoff | **~200 MB/s** — consistent |
+| Random write | Terrible (shingle rewrite penalty on every random write) | Good (enterprise-class) |
+| Concurrent R+W | Poor (SMR write amplification under mixed load) | Fine |
+| Typical use | Cheap consumer bulk storage | Any write-heavy workload |
+
+### How to Identify SMR vs CMR
+
+| Brand | SMR Models | CMR Models |
+|-------|-----------|------------|
+| Seagate | Barracuda Compute (STx000DM00x), most 2.5" | IronWolf Pro, Exos, Enterprise |
+| WD | WD Blue, WD Green (certain sizes) | WD Red Plus, Red Pro, Gold, Ultrastar |
+| HGST | (none — all HGST drives are CMR, mostly helium) | All models, incl. He8/He10/He12 |
+
+**Reliable rule:** Enterprise/server-class drives (Ultrastar, Exos, IronWolf Pro, WD Gold, Seagate Exos) are always CMR. Consumer "value" lines (Barracuda Compute, WD Blue/Green) are often SMR after certain capacities.
+
+### Where SMR Actually Hurts in a Homelab
+
+| Workload | SMR Impact | CMR Impact |
+|----------|-----------|------------|
+| Plex/Jellyfin direct stream | None — reads only | Same |
+| Immich photo/video import | Significant — writes slow down after a few GB | Fast, consistent |
+| Large file copy (>10GB) | Noticeable — starts fast, chokes | Fast throughout |
+| Server backup (rsync) | Significant — long tail on large datasets | Predictable speed |
+| Docker database storage | Painful — random writes trigger constant shingle rewrites | Fine |
+| Photo library browsing | None — reads only | Same |
+
+**Verdict:** For a media server that mostly reads (Plex), SMR is fine. For anything that writes regularly (Immich, database storage, backup target, photo/video editing working drive), CMR is worth the premium.
+
+## Speed Tiers for HDDs
+
+| Tier | RPM | Tech | Seq Read | Sustained Write | Use Case |
+|------|-----|------|----------|----------------|----------|
+| Consumer SMR | 5400 | SMR | ~150 MB/s | ~30-50 MB/s | Cheap cold storage, write-once media |
+| Consumer CMR | 5400-7200 | CMR | ~180 MB/s | ~150-180 MB/s | General bulk storage, mixed workloads |
+| Enterprise helium | 7200 | CMR | ~200-210 MB/s | ~195-200 MB/s | Active storage, Immich, databases, heavy writes |
+| Enterprise SAS | 10K-15K | CMR | ~150-250 MB/s | ~150-250 MB/s | Legacy database tier (obsolete vs SSD) |
+| SATA SSD | N/A | NAND | ~500 MB/s | ~450 MB/s | Active containers, DB, OS |
+| NVMe | N/A | NAND | 2-7 GB/s | 1-6 GB/s | Boot, heavy DB, compute |
+
+**Real-world impact:** Going from a 5400 SMR consumer drive to a 7200 CMR enterprise helium drive gives ~30-40% faster sequential reads and **3-5x faster sustained writes**. For homelab use, the biggest real-world gains are during large media imports/transfers and concurrent R+W (Immich thumbnailing while uploading).
+
+## Checking Your Current Drive's Specs
+
+```bash
+# Model name + RPM hint (RPM not always reported)
+lsblk -o NAME,SIZE,MODEL,TRAN,MOUNTPOINT
+
+# Detailed SMART info — look for RPM, rotation rate
+sudo smartctl -a /dev/sdX | grep -iE "rotation rate|rpm|form factor|sector size"
+
+# Confirm SMR vs CMR by model number lookup or teardown review (no reliable OS-level check)
+```
+
+Note: `smartctl` may not report RPM for USB-attached drives behind SATA bridges.
+
+## RAM Upgrade Compatibility (OEM Systems)
+
+When the user asks about RAM upgrades, especially on HP, Dell, or Lenovo OEM desktop systems:
+
+### Key Constraints
+
+| Factor | What to check |
+|--------|--------------|
+| Max capacity | `sudo dmidecode --type memory \| grep -i "Maximum Capacity"` |
+| DIMM slots | `sudo dmidecode --type memory \| grep -c "Memory Device"` |
+| Current config | `sudo dmidecode --type memory \| grep -E "Speed|Part Number|Configured"` |
+| XMP support | Check if configured speed > JEDEC (2133/2400 for DDR4, 4800/5600 for DDR5) — higher speed means XMP is working |
+| CPU generation | `cat /proc/cpuinfo \| grep "model name" \| head -1` — dictates IMC speed ceiling |
+
+### OEM BIOS XMP Likelihood
+
+| OEM | XMP Support |
+|-----|------------|
+| **HP OMEN** (gaming line) | 🟢 Good — HP enables overclocked speeds (3200 confirmed on HP 8703 with i7-10700K) |
+| **HP Pro/Elite** (business line) | 🔴 Rare — locked to JEDEC, no XMP |
+| **Dell XPS/Gaming** | 🟡 Mixed — some support, some locked |
+| **Dell Optiplex** | 🔴 Almost never — locked BIOS |
+| **Lenovo Legion** | 🟢 Good — similar to OMEN gaming line |
+| **Lenovo ThinkCentre** | 🔴 Locked — JEDEC only |
+| **Custom/DIY** | 🟢 Always — any consumer motherboard supports XMP |
+
+### Speed Expectations
+
+- **If current RAM runs above JEDEC** (e.g., DDR4-3200 on a Comet Lake system whose JEDEC max is 2933): XMP works. Higher-speed kits (3600-3866) will likely work or fall back gracefully.
+- **If current RAM runs at JEDEC** (2133/2400/2933 for DDR4): the BIOS may not support XMP at all. A 3600 kit will still work, but at JEDEC speed (~2400-2933).
+- **If the kit doesn't POST at its rated speed:** the board will fall back to JEDEC SPD timings. The user still gets the capacity upgrade.
+
+### Real-World Performance
+
+| Speed Difference | Gaming Perf | File Server Perf | Docker/Containers |
+|-----------------|------------|-------------------|-------------------|
+| 3200 → 3600 | ≤3% | Not noticeable | Not noticeable |
+| 2133 → 3200 | 8-12% | Minimal | Slightly snappier for CPU-bound workloads |
+| 16GB → 32GB | 0% (unless maxed out) | Noticeable with many containers | **Significant** — more room for containers, RAM cache |
+| 32GB → 64GB | 0% | Only if running VMs | Only if running heavy DB workloads |
+
+**The capacity upgrade (16→32GB) is almost always more impactful than the speed bump (3200→3600 MHz) for server workloads.**
@@ -0,0 +1,70 @@
+# Samba NAS — Session Config (rayserver)
+
+Deployed on `rayserver` (Ubuntu 26.04, 192.168.50.98) for three drives.
+
+## Shares
+
+| Share | Mount Point | Drive | Filesystem | Size |
+|-------|-------------|-------|------------|------|
+| `media` | /mnt/media | SanDisk Extreme 1TB (USB SSD) | exfat | 932G, ~54% used |
+| `storage` | /mnt/storage | WD Blue 1TB 7200RPM HDD (WD10EZEX, SATA) | ext4 | 916G, ~1% used |
+| `wd-passport` | /mnt/wd-passport | WD Passport 2.7TB (USB HDD) | ntfs-3g | 2.8T, ~31% used |
+
+## Speed Benchmarks
+
+| Drive | Read | Write | Connection |
+|-------|------|-------|-----------|
+| NVMe boot (WD Black SN530) | — | — | NVMe (fastest) |
+| sda (SanDisk Extreme USB) | — | — | USB 3.1 Gen2 10Gbps |
+| **sdb (WD Blue HDD SATA)** | **~192 MB/s** | **~156 MB/s** | SATA 3 |
+| **sdc (WD Passport USB)** | **~54 MB/s** | **~49 MB/s** | USB 3.0 5Gbps |
+
+sdb is ~3x faster than sdc due to direct SATA vs USB bridge bottleneck.
+
+## smb.conf (/etc/samba/smb.conf)
+
+```ini
+[global]
+workgroup = WORKGROUP
+server string = rayserver
+netbios name = rayserver
+security = user
+map to guest = Bad User
+guest account = nobody
+server min protocol = SMB2
+client min protocol = SMB2
+
+[media]
+path = /mnt/media
+browseable = yes
+read only = no
+guest ok = yes
+force user = ray
+create mask = 0777
+directory mask = 0777
+
+[storage]
+path = /mnt/storage
+browseable = yes
+read only = no
+guest ok = yes
+force user = ray
+create mask = 0777
+directory mask = 0777
+
+[wd-passport]
+path = /mnt/wd-passport
+browseable = yes
+read only = no
+guest ok = yes
+force user = ray
+create mask = 0777
+directory mask = 0777
+```
+
+## Client Access
+
+- **Windows**: `\\192.168.50.98` or `\\rayserver`
+- **macOS**: `smb://192.168.50.98`
+- **Linux**: `smb://192.168.50.98/`
+- No password (guest access, trusted LAN)
@@ -0,0 +1,229 @@
+---
+name: data-migration
+description: "Disk-to-disk data migration and backup — rsync best practices, USB I/O considerations, permission handling, verification."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [backup, migration, rsync, storage, data-transfer]
+    related_skills: [server-health-check]
+---
+
+# Data Migration & Disk Backup
+
+## Overview
+
+Local disk-to-disk backup and data migration using rsync on Linux. Covers the hard-won lessons about USB drive I/O, the `-W` flag, permission handling, and verification.
+
+## General Principle: Think First, Execute Second
+
+**Before running any backup command, stop and evaluate the most efficient approach.** Don't default to the first command that comes to mind. Consider:
+
+- **What's the bottleneck?** USB bus? Source drive speed? Destination write speed?
+- **What's the fastest approach?** Is whole-file (`-W`) appropriate? Would compression help or hurt? Is the source on USB (no parallel I/O)?
+- **Are there better flags?** The default `-avhP` computes checksums for delta transfer — wasteful on local copies where `-avhW` is dramatically faster.
+- **Could the plan be wrong?** If the user suggests something you already tried, explain why it failed rather than blindly re-executing.
+
+Take 10 seconds to think through the alternatives before writing the first rsync command. Picking the right flags upfront saves hours of rework.
+
+## User Preference: Plan Before Executing
+
+This user wants you to **stop and think** before running commands. When asked to do a task, evaluate the most efficient approach first — don't default to option A when option C is 10× faster. Consider the bottleneck (USB bus, source read, dest write), the right flags (`-W` over delta), and whether the plan could fail. Getting the approach right upfront saves hours.
+
+## Key Lessons (from real failures)
+
+### 1. Use `-W` (whole-file) for local disk copies
+
+**Don't use:** `rsync -avhP` — this computes block-level checksums for delta transfer. Great for networks. **Wasteful for local disk-to-disk** — it reads every block on both sides.
+
+**Do use:** `rsync -avhW` — streams the entire file without checksumming. The `-W` flag skips the delta algorithm and just copies. For local USB-to-SATA copies, this is dramatically faster.
+
+**Progress display:** Prefer `--info=progress2` over `--progress` for large transfers (100K+ files). `--progress` outputs per-file progress lines that flood the log; `--info=progress2` prints a single summary line that updates in-place via `\r`, keeping output manageable — critical when debugging errors buried in thousands of lines.
+
+```
+Bad:   rsync -avhP /src/ /dst/        # checksums everything — slow
+Good:  rsync -avhW /src/ /dst/        # whole-file copy — fast
+```
+
+### 2. USB drives cannot handle parallel transfers
+
+**Don't** run parallel rsync/processes against a single USB drive. The drive's controller + USB bus create a single I/O channel. Multiple readers/writers cause **I/O contention** — processes hang in D state (uninterruptible sleep), throughput collapses, and processes eventually get killed.
+
+**Do** use a single sequential rsync. It will be slower than a SATA copy but faster than a contended one.
+
+```
+Bad:   rsync ... & rsync ... & rsync ... &   # 3 processes = hang
+Good:  rsync -avhW /src/ /dst/                # 1 process = steady
+```
+
+If you need background I/O priority, use `ionice`:
+```
+For background runs that survive SSH disconnection, use `ionice` with **best-effort low** priority:
+
+```bash
+nohup sudo ionice -c 2 -n 7 rsync -avhW --progress /src/ /dst/ \
+  --exclude='$RECYCLE.BIN' --exclude='System Volume Information' \
+  > /tmp/backup.log 2>&1 &
+```
+
+**Why `-c 2 -n 7` and not `-c 3` (idle):** Idle scheduling can starve completely on a busy system — the process never gets I/O time and makes zero progress. Best-effort low priority still gets a scheduling slice and keeps moving steadily, while yielding to higher-priority I/O.
+
+If you run without `nohup` and the SSH session disconnects, rsync gets SIGHUP and dies. Always use `nohup` + log redirection + `&` when running over SSH.
+
+Monitor with:
+```bash
+ps aux | grep rsync         # alive? (R = good, S = normal, D = stuck)
+df -h /dst/                 # fastest progress check — no permission-denied errors
+tail -c 500 /tmp/log | strings | grep -oP '\d+\.\d+MB/s' | tail -3   # current speed
+```
+
+**Note on log parsing:** rsync uses `\r` (carriage returns) for progress lines, so plain `tail` shows garbled lines. Use `tail -c 500 | strings` to extract readable text.
+
+**Completion verification:** After rsync exits, confirm with a dry-run + log check:
+```bash
+# No rsync process running? Verify nothing was missed:
+sudo rsync -avhWn /src/ /dst/ --exclude=... 2>&1 | tail -3
+# Clean output (no file list) = fully synced
+```
+
+### 3. Run rsync as root (sudo) when source has mixed ownership
+
+NTFS/exFAT drives mounted by a normal user show all files as owned by the user. But if any files on the source have restricted permissions (e.g. Docker volumes owned by UID 999), rsync as a regular user will hit "Permission denied" errors on the destination.
+
+**Always use `sudo rsync` for full-disk backups** that include system or container data. The destination will have root-owned files, which is fine for a backup drive.
+
+### 4. Exclude junk upfront
+
+NTFS/exFAT drives accumulate junk directories. Exclude them explicitly:
+
+```
+--exclude='$RECYCLE.BIN'
+--exclude='System Volume Information'
+--exclude='Recovered data*'
+--exclude='msdia80.dll'
+```
+
+### 5. Verify with dry-run after completion
+
+After the main rsync finishes, run a dry-run to confirm nothing was missed:
+
+```
+sudo rsync -avhWn /mnt/source/ /mnt/dest/ --exclude='$RECYCLE.BIN' ...
+```
+
+A clean dry-run outputs only directory paths (no files) because everything is already in sync. If it lists files, those still need copying.
+
+## Procedure
+
+### Step 1: Survey the landscape
+
+```bash
+# Source: total used space (includes junk)
+df -h /mnt/source/
+
+# Source: actual data to copy (excl junk)
+sudo du -sh /mnt/source/ --exclude='$RECYCLE.BIN' --exclude='System Volume Information'
+
+# Destination: available space
+df -h /mnt/dest/
+```
+
+### Step 2: Run the backup
+
+```bash
+sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-name/ \
+  --exclude='$RECYCLE.BIN' \
+  --exclude='System Volume Information' \
+  --exclude='Recovered data*' \
+  --exclude='msdia80.dll'
+```
+
+### Step 3: Verify
+
+```bash
+# Check size matches
+sudo du -sh /mnt/dest/backup-name/
+
+# Dry-run to confirm no remaining files
+sudo rsync -avhWn /mnt/source/ /mnt/dest/backup-name/ \
+  --exclude='$RECYCLE.BIN' --exclude='System Volume Information'
+```
+
+### Step 4: Monitor
+
+If running in background, check periodically:
+```bash
+ps aux | grep rsync
+# D state = stuck in disk I/O (bad sign if prolonged)
+# S state = sleeping/waiting (normal)
+# R state = actively reading/writing (good)
+```
+
+If stuck in D state for >5 minutes with no progress, the process is likely hung on USB I/O contention. Kill and restart with a single instance.
+
+## Post-Backup Reorganization
+
+After a backup completes, users often want the data at the root of the destination drive rather than nested in a subdirectory:
+
+```bash
+# Before: /mnt/dest/backup-name/Photos/, /mnt/dest/backup-name/Videos/
+# After:  /mnt/dest/Photos/, /mnt/dest/Videos/
+
+# Move everything out of the subdirectory (same filesystem = instant, no data copy)
+sudo mv /mnt/dest/backup-name/* /mnt/dest/
+sudo mv /mnt/dest/backup-name/.* /mnt/dest/ 2>/dev/null   # hidden files
+sudo rmdir /mnt/dest/backup-name/
+
+# Optionally clean up junk that snuck through from early rsync runs
+sudo rm -rf "/mnt/dest/Recovered data*" "/mnt/dest/System Volume Information" /mnt/dest/msdia80.dll
+```
+
+### Merging recovered data into existing directories
+
+Data recovery tools (like those found in `Deep Scan result/` folders) often organize recovered files by camera make/model or file type. Users commonly want these merged into their existing organized library:
+
+```bash
+# Before merge:
+# /mnt/dest/Photos/ (existing backup)
+# /mnt/dest/Deep Scan result/Photos_deepscan/Camera/ (recovered photos)
+# /mnt/dest/Deep Scan result/Videos/More Lost Files(RAW)/ (recovered videos)
+
+# 1. Separate photo and video folders within the recovered data
+mkdir -p "Deep Scan result/Photos" "Deep Scan result/Videos"
+mv "Deep Scan result/Photos_deepscan/Camera" "Deep Scan result/Photos/"
+mv "Deep Scan result/Videos More Lost Files(RAW)" "Deep Scan result/Videos/"
+mv "Deep Scan result/Videos Mov" "Deep Scan result/Videos/"
+
+# 2. Merge into main directories
+mv "Deep Scan result/Photos/Camera" /mnt/dest/Photos/
+mv "Deep Scan result/Videos/More Lost Files(RAW)" /mnt/dest/Videos/
+mv "Deep Scan result/Videos/Mov" /mnt/dest/Videos/
+```
+
+This is always instant (same-filesystem moves, no data copying). Only directory metadata is updated.
+
+## Performance Expectations
+
+File type dramatically affects transfer speed on USB drives:
+
+| File Type | Typical Speed | Why |
+|-----------|--------------|-----|
+| Large videos (500MB+) | **75–100 MB/s** | Sequential reads, minimal metadata overhead |
+| Photos (2-10MB) | **40–60 MB/s** | Mixed sequential/random |
+| Small files (<1MB, thumbnails, metadata) | **15–40 MB/s** | Directory creation, metadata overhead, random I/O |
+| Mixed (full drive backup) | **50–75 MB/s avg** | Depends on file size distribution |
+
+**Expect the tail to slow down.** After large video files finish, the remaining small files (Immich thumbnails, library metadata) will drop to 15-40 MB/s. A 642 GB backup might take ~1.5-2 hours despite the first 500 GB flying through.
+
+## Pitfalls
+
+- **D state panic:** One or more rsync processes in D state (uninterruptible sleep) for extended periods usually means USB I/O contention. Kill them all and restart with a single sequential instance.
+- **Permission denied on destination:** First rsync run without sudo creates root-owned directories. Subsequent runs as user fail. Use `sudo` consistently, or `chown` the dest after.
+- **Silent permission-denied pattern (diagnostic):** When rsync produces thousands of progress lines (`to-chk` counts down from N to 0) but transfers 0% with `xfr#0` and exits code 23, the mkdir failed at the very start. The actual error (`recv_generator: mkdir "/dest/dir" failed: Permission denied (13)`) only appears ONCE near the top of the output, buried among 2800+ progress lines. The fix is `sudo chown user:user /mnt/dest/` (or `sudo rsync`). Don't waste time reading pages of progress — grep for `denied` or `error` first.
+- **Space miscalculation:** ext4 reserves 5% of blocks for root (default). On an 8TB drive, that's ~372GB "missing" from what the user expects. Check with `tune2fs -l /dev/sdX1 | grep Reserved` and explain decimal-vs-binary if questioned.
+- **Partial completion:** If rsync exits/crashes (common with USB), the next run with the same flags is incremental — it only copies what's missing. No need to start over.
+- **WATCH OUT for stale temp files:** Chrome `.crdownload` and `part*.tmp` files from interrupted downloads waste space and confuse size estimates. Clean them before estimating disk usage.
+- **SSH quoting of `$` in exclude patterns:** When running rsync via `ssh host 'nohup sudo rsync ... --exclude='\''$RECYCLE.BIN'\'' ...'`, the `$` in `$RECYCLE.BIN` requires careful shell quoting. The `'\''...'\''` pattern (break out of outer single quotes, insert literal `'`, re-enter single quotes) works but is fragile. **Safer alternative:** Write the full command to a script on the remote server first, or use a heredoc-style variable on the remote side. A missed `$` expansion means the exclude silently becomes `--exclude=.BIN` (empty variable), and the junk folder gets copied.
+- **`du` is slow on permission-heavy dirs:** Using `sudo du -sh /mnt/dest/backup/` to check progress is slow (minutes) when the dest has thousands of Immich-style hex-nested directories with mixed Docker permissions. **Prefer `df -h /mnt/dest/` for a fast byte-level snapshot** — it shows used space on the whole filesystem, which is accurate enough for progress monitoring. Reserve `du` for final verification.