hermes-config/skills/devops/storage-management/references/rsync-best-practices.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
data-migration	Disk-to-disk data migration and backup — rsync best practices, USB I/O considerations, permission handling, verification.	1.0.0	Hermes Agent	MIT	linux	hermes tags related_skills backup migration rsync storage data-transfer server-health-check

Data Migration & Disk Backup

Overview

Local disk-to-disk backup and data migration using rsync on Linux. Covers the hard-won lessons about USB drive I/O, the -W flag, permission handling, and verification.

General Principle: Think First, Execute Second

Before running any backup command, stop and evaluate the most efficient approach. Don't default to the first command that comes to mind. Consider:

• What's the bottleneck? USB bus? Source drive speed? Destination write speed?
• What's the fastest approach? Is whole-file (-W) appropriate? Would compression help or hurt? Is the source on USB (no parallel I/O)?
• Are there better flags? The default -avhP computes checksums for delta transfer — wasteful on local copies where -avhW is dramatically faster.
• Could the plan be wrong? If the user suggests something you already tried, explain why it failed rather than blindly re-executing.

Take 10 seconds to think through the alternatives before writing the first rsync command. Picking the right flags upfront saves hours of rework.

User Preference: Plan Before Executing

This user wants you to stop and think before running commands. When asked to do a task, evaluate the most efficient approach first — don't default to option A when option C is 10× faster. Consider the bottleneck (USB bus, source read, dest write), the right flags (-W over delta), and whether the plan could fail. Getting the approach right upfront saves hours.

Key Lessons (from real failures)

1. Use -W (whole-file) for local disk copies

Don't use: rsync -avhP — this computes block-level checksums for delta transfer. Great for networks. Wasteful for local disk-to-disk — it reads every block on both sides.

Do use: rsync -avhW — streams the entire file without checksumming. The -W flag skips the delta algorithm and just copies. For local USB-to-SATA copies, this is dramatically faster.

Progress display: Prefer --info=progress2 over --progress for large transfers (100K+ files). --progress outputs per-file progress lines that flood the log; --info=progress2 prints a single summary line that updates in-place via \r, keeping output manageable — critical when debugging errors buried in thousands of lines.

Bad:   rsync -avhP /src/ /dst/        # checksums everything — slow
Good:  rsync -avhW /src/ /dst/        # whole-file copy — fast

2. USB drives cannot handle parallel transfers

Don't run parallel rsync/processes against a single USB drive. The drive's controller + USB bus create a single I/O channel. Multiple readers/writers cause I/O contention — processes hang in D state (uninterruptible sleep), throughput collapses, and processes eventually get killed.

Do use a single sequential rsync. It will be slower than a SATA copy but faster than a contended one.

Bad:   rsync ... & rsync ... & rsync ... &   # 3 processes = hang
Good:  rsync -avhW /src/ /dst/                # 1 process = steady

If you need background I/O priority, use ionice:

For background runs that survive SSH disconnection, use `ionice` with **best-effort low** priority:

```bash
nohup sudo ionice -c 2 -n 7 rsync -avhW --progress /src/ /dst/ \
  --exclude='$RECYCLE.BIN' --exclude='System Volume Information' \
  > /tmp/backup.log 2>&1 &

Why -c 2 -n 7 and not -c 3 (idle): Idle scheduling can starve completely on a busy system — the process never gets I/O time and makes zero progress. Best-effort low priority still gets a scheduling slice and keeps moving steadily, while yielding to higher-priority I/O.

If you run without nohup and the SSH session disconnects, rsync gets SIGHUP and dies. Always use nohup + log redirection + & when running over SSH.

Monitor with:

ps aux | grep rsync         # alive? (R = good, S = normal, D = stuck)
df -h /dst/                 # fastest progress check — no permission-denied errors
tail -c 500 /tmp/log | strings | grep -oP '\d+\.\d+MB/s' | tail -3   # current speed

Note on log parsing: rsync uses \r (carriage returns) for progress lines, so plain tail shows garbled lines. Use tail -c 500 | strings to extract readable text.

Completion verification: After rsync exits, confirm with a dry-run + log check:

# No rsync process running? Verify nothing was missed:
sudo rsync -avhWn /src/ /dst/ --exclude=... 2>&1 | tail -3
# Clean output (no file list) = fully synced

3. Run rsync as root (sudo) when source has mixed ownership

NTFS/exFAT drives mounted by a normal user show all files as owned by the user. But if any files on the source have restricted permissions (e.g. Docker volumes owned by UID 999), rsync as a regular user will hit "Permission denied" errors on the destination.

Always use sudo rsync for full-disk backups that include system or container data. The destination will have root-owned files, which is fine for a backup drive.

4. Exclude junk upfront

NTFS/exFAT drives accumulate junk directories. Exclude them explicitly:

--exclude='$RECYCLE.BIN'
--exclude='System Volume Information'
--exclude='Recovered data*'
--exclude='msdia80.dll'

5. Verify with dry-run after completion

After the main rsync finishes, run a dry-run to confirm nothing was missed:

sudo rsync -avhWn /mnt/source/ /mnt/dest/ --exclude='$RECYCLE.BIN' ...

A clean dry-run outputs only directory paths (no files) because everything is already in sync. If it lists files, those still need copying.

Procedure

Step 1: Survey the landscape

# Source: total used space (includes junk)
df -h /mnt/source/

# Source: actual data to copy (excl junk)
sudo du -sh /mnt/source/ --exclude='$RECYCLE.BIN' --exclude='System Volume Information'

# Destination: available space
df -h /mnt/dest/

Step 2: Run the backup

sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-name/ \
  --exclude='$RECYCLE.BIN' \
  --exclude='System Volume Information' \
  --exclude='Recovered data*' \
  --exclude='msdia80.dll'

Step 3: Verify

# Check size matches
sudo du -sh /mnt/dest/backup-name/

# Dry-run to confirm no remaining files
sudo rsync -avhWn /mnt/source/ /mnt/dest/backup-name/ \
  --exclude='$RECYCLE.BIN' --exclude='System Volume Information'

Step 4: Monitor

If running in background, check periodically:

ps aux | grep rsync
# D state = stuck in disk I/O (bad sign if prolonged)
# S state = sleeping/waiting (normal)
# R state = actively reading/writing (good)

If stuck in D state for >5 minutes with no progress, the process is likely hung on USB I/O contention. Kill and restart with a single instance.

Post-Backup Reorganization

After a backup completes, users often want the data at the root of the destination drive rather than nested in a subdirectory:

# Before: /mnt/dest/backup-name/Photos/, /mnt/dest/backup-name/Videos/
# After:  /mnt/dest/Photos/, /mnt/dest/Videos/

# Move everything out of the subdirectory (same filesystem = instant, no data copy)
sudo mv /mnt/dest/backup-name/* /mnt/dest/
sudo mv /mnt/dest/backup-name/.* /mnt/dest/ 2>/dev/null   # hidden files
sudo rmdir /mnt/dest/backup-name/

# Optionally clean up junk that snuck through from early rsync runs
sudo rm -rf "/mnt/dest/Recovered data*" "/mnt/dest/System Volume Information" /mnt/dest/msdia80.dll

Merging recovered data into existing directories

Data recovery tools (like those found in Deep Scan result/ folders) often organize recovered files by camera make/model or file type. Users commonly want these merged into their existing organized library:

# Before merge:
# /mnt/dest/Photos/ (existing backup)
# /mnt/dest/Deep Scan result/Photos_deepscan/Camera/ (recovered photos)
# /mnt/dest/Deep Scan result/Videos/More Lost Files(RAW)/ (recovered videos)

# 1. Separate photo and video folders within the recovered data
mkdir -p "Deep Scan result/Photos" "Deep Scan result/Videos"
mv "Deep Scan result/Photos_deepscan/Camera" "Deep Scan result/Photos/"
mv "Deep Scan result/Videos More Lost Files(RAW)" "Deep Scan result/Videos/"
mv "Deep Scan result/Videos Mov" "Deep Scan result/Videos/"

# 2. Merge into main directories
mv "Deep Scan result/Photos/Camera" /mnt/dest/Photos/
mv "Deep Scan result/Videos/More Lost Files(RAW)" /mnt/dest/Videos/
mv "Deep Scan result/Videos/Mov" /mnt/dest/Videos/

This is always instant (same-filesystem moves, no data copying). Only directory metadata is updated.

Performance Expectations

File type dramatically affects transfer speed on USB drives:

File Type	Typical Speed	Why
Large videos (500MB+)	75–100 MB/s	Sequential reads, minimal metadata overhead
Photos (2-10MB)	40–60 MB/s	Mixed sequential/random
Small files (<1MB, thumbnails, metadata)	15–40 MB/s	Directory creation, metadata overhead, random I/O
Mixed (full drive backup)	50–75 MB/s avg	Depends on file size distribution

Expect the tail to slow down. After large video files finish, the remaining small files (Immich thumbnails, library metadata) will drop to 15-40 MB/s. A 642 GB backup might take ~1.5-2 hours despite the first 500 GB flying through.

Pitfalls

• D state panic: One or more rsync processes in D state (uninterruptible sleep) for extended periods usually means USB I/O contention. Kill them all and restart with a single sequential instance.
• Permission denied on destination: First rsync run without sudo creates root-owned directories. Subsequent runs as user fail. Use sudo consistently, or chown the dest after.
• Silent permission-denied pattern (diagnostic): When rsync produces thousands of progress lines (to-chk counts down from N to 0) but transfers 0% with xfr#0 and exits code 23, the mkdir failed at the very start. The actual error (recv_generator: mkdir "/dest/dir" failed: Permission denied (13)) only appears ONCE near the top of the output, buried among 2800+ progress lines. The fix is sudo chown user:user /mnt/dest/ (or sudo rsync). Don't waste time reading pages of progress — grep for denied or error first.
• Space miscalculation: ext4 reserves 5% of blocks for root (default). On an 8TB drive, that's ~372GB "missing" from what the user expects. Check with tune2fs -l /dev/sdX1 | grep Reserved and explain decimal-vs-binary if questioned.
• Partial completion: If rsync exits/crashes (common with USB), the next run with the same flags is incremental — it only copies what's missing. No need to start over.
• WATCH OUT for stale temp files: Chrome .crdownload and part*.tmp files from interrupted downloads waste space and confuse size estimates. Clean them before estimating disk usage.
• SSH quoting of $ in exclude patterns: When running rsync via ssh host 'nohup sudo rsync ... --exclude='\''$RECYCLE.BIN'\'' ...', the $ in $RECYCLE.BIN requires careful shell quoting. The '\''...'\'' pattern (break out of outer single quotes, insert literal ', re-enter single quotes) works but is fragile. Safer alternative: Write the full command to a script on the remote server first, or use a heredoc-style variable on the remote side. A missed $ expansion means the exclude silently becomes --exclude=.BIN (empty variable), and the junk folder gets copied.
• du is slow on permission-heavy dirs: Using sudo du -sh /mnt/dest/backup/ to check progress is slow (minutes) when the dest has thousands of Immich-style hex-nested directories with mixed Docker permissions. Prefer df -h /mnt/dest/ for a fast byte-level snapshot — it shows used space on the whole filesystem, which is accurate enough for progress monitoring. Reserve du for final verification.