ray
230 lines
12 KiB
Markdown
230 lines
12 KiB
Markdown
---
|
||
name: data-migration
|
||
description: "Disk-to-disk data migration and backup — rsync best practices, USB I/O considerations, permission handling, verification."
|
||
version: 1.0.0
|
||
author: Hermes Agent
|
||
license: MIT
|
||
platforms: [linux]
|
||
metadata:
|
||
hermes:
|
||
tags: [backup, migration, rsync, storage, data-transfer]
|
||
related_skills: [server-health-check]
|
||
---
|
||
|
||
# Data Migration & Disk Backup
|
||
|
||
## Overview
|
||
|
||
Local disk-to-disk backup and data migration using rsync on Linux. Covers the hard-won lessons about USB drive I/O, the `-W` flag, permission handling, and verification.
|
||
|
||
## General Principle: Think First, Execute Second
|
||
|
||
**Before running any backup command, stop and evaluate the most efficient approach.** Don't default to the first command that comes to mind. Consider:
|
||
|
||
- **What's the bottleneck?** USB bus? Source drive speed? Destination write speed?
|
||
- **What's the fastest approach?** Is whole-file (`-W`) appropriate? Would compression help or hurt? Is the source on USB (no parallel I/O)?
|
||
- **Are there better flags?** The default `-avhP` computes checksums for delta transfer — wasteful on local copies where `-avhW` is dramatically faster.
|
||
- **Could the plan be wrong?** If the user suggests something you already tried, explain why it failed rather than blindly re-executing.
|
||
|
||
Take 10 seconds to think through the alternatives before writing the first rsync command. Picking the right flags upfront saves hours of rework.
|
||
|
||
## User Preference: Plan Before Executing
|
||
|
||
This user wants you to **stop and think** before running commands. When asked to do a task, evaluate the most efficient approach first — don't default to option A when option C is 10× faster. Consider the bottleneck (USB bus, source read, dest write), the right flags (`-W` over delta), and whether the plan could fail. Getting the approach right upfront saves hours.
|
||
|
||
## Key Lessons (from real failures)
|
||
|
||
### 1. Use `-W` (whole-file) for local disk copies
|
||
|
||
**Don't use:** `rsync -avhP` — this computes block-level checksums for delta transfer. Great for networks. **Wasteful for local disk-to-disk** — it reads every block on both sides.
|
||
|
||
**Do use:** `rsync -avhW` — streams the entire file without checksumming. The `-W` flag skips the delta algorithm and just copies. For local USB-to-SATA copies, this is dramatically faster.
|
||
|
||
**Progress display:** Prefer `--info=progress2` over `--progress` for large transfers (100K+ files). `--progress` outputs per-file progress lines that flood the log; `--info=progress2` prints a single summary line that updates in-place via `\r`, keeping output manageable — critical when debugging errors buried in thousands of lines.
|
||
|
||
```
|
||
Bad: rsync -avhP /src/ /dst/ # checksums everything — slow
|
||
Good: rsync -avhW /src/ /dst/ # whole-file copy — fast
|
||
```
|
||
|
||
### 2. USB drives cannot handle parallel transfers
|
||
|
||
**Don't** run parallel rsync/processes against a single USB drive. The drive's controller + USB bus create a single I/O channel. Multiple readers/writers cause **I/O contention** — processes hang in D state (uninterruptible sleep), throughput collapses, and processes eventually get killed.
|
||
|
||
**Do** use a single sequential rsync. It will be slower than a SATA copy but faster than a contended one.
|
||
|
||
```
|
||
Bad: rsync ... & rsync ... & rsync ... & # 3 processes = hang
|
||
Good: rsync -avhW /src/ /dst/ # 1 process = steady
|
||
```
|
||
|
||
If you need background I/O priority, use `ionice`:
|
||
```
|
||
For background runs that survive SSH disconnection, use `ionice` with **best-effort low** priority:
|
||
|
||
```bash
|
||
nohup sudo ionice -c 2 -n 7 rsync -avhW --progress /src/ /dst/ \
|
||
--exclude='$RECYCLE.BIN' --exclude='System Volume Information' \
|
||
> /tmp/backup.log 2>&1 &
|
||
```
|
||
|
||
**Why `-c 2 -n 7` and not `-c 3` (idle):** Idle scheduling can starve completely on a busy system — the process never gets I/O time and makes zero progress. Best-effort low priority still gets a scheduling slice and keeps moving steadily, while yielding to higher-priority I/O.
|
||
|
||
If you run without `nohup` and the SSH session disconnects, rsync gets SIGHUP and dies. Always use `nohup` + log redirection + `&` when running over SSH.
|
||
|
||
Monitor with:
|
||
```bash
|
||
ps aux | grep rsync # alive? (R = good, S = normal, D = stuck)
|
||
df -h /dst/ # fastest progress check — no permission-denied errors
|
||
tail -c 500 /tmp/log | strings | grep -oP '\d+\.\d+MB/s' | tail -3 # current speed
|
||
```
|
||
|
||
**Note on log parsing:** rsync uses `\r` (carriage returns) for progress lines, so plain `tail` shows garbled lines. Use `tail -c 500 | strings` to extract readable text.
|
||
|
||
**Completion verification:** After rsync exits, confirm with a dry-run + log check:
|
||
```bash
|
||
# No rsync process running? Verify nothing was missed:
|
||
sudo rsync -avhWn /src/ /dst/ --exclude=... 2>&1 | tail -3
|
||
# Clean output (no file list) = fully synced
|
||
```
|
||
|
||
### 3. Run rsync as root (sudo) when source has mixed ownership
|
||
|
||
NTFS/exFAT drives mounted by a normal user show all files as owned by the user. But if any files on the source have restricted permissions (e.g. Docker volumes owned by UID 999), rsync as a regular user will hit "Permission denied" errors on the destination.
|
||
|
||
**Always use `sudo rsync` for full-disk backups** that include system or container data. The destination will have root-owned files, which is fine for a backup drive.
|
||
|
||
### 4. Exclude junk upfront
|
||
|
||
NTFS/exFAT drives accumulate junk directories. Exclude them explicitly:
|
||
|
||
```
|
||
--exclude='$RECYCLE.BIN'
|
||
--exclude='System Volume Information'
|
||
--exclude='Recovered data*'
|
||
--exclude='msdia80.dll'
|
||
```
|
||
|
||
### 5. Verify with dry-run after completion
|
||
|
||
After the main rsync finishes, run a dry-run to confirm nothing was missed:
|
||
|
||
```
|
||
sudo rsync -avhWn /mnt/source/ /mnt/dest/ --exclude='$RECYCLE.BIN' ...
|
||
```
|
||
|
||
A clean dry-run outputs only directory paths (no files) because everything is already in sync. If it lists files, those still need copying.
|
||
|
||
## Procedure
|
||
|
||
### Step 1: Survey the landscape
|
||
|
||
```bash
|
||
# Source: total used space (includes junk)
|
||
df -h /mnt/source/
|
||
|
||
# Source: actual data to copy (excl junk)
|
||
sudo du -sh /mnt/source/ --exclude='$RECYCLE.BIN' --exclude='System Volume Information'
|
||
|
||
# Destination: available space
|
||
df -h /mnt/dest/
|
||
```
|
||
|
||
### Step 2: Run the backup
|
||
|
||
```bash
|
||
sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-name/ \
|
||
--exclude='$RECYCLE.BIN' \
|
||
--exclude='System Volume Information' \
|
||
--exclude='Recovered data*' \
|
||
--exclude='msdia80.dll'
|
||
```
|
||
|
||
### Step 3: Verify
|
||
|
||
```bash
|
||
# Check size matches
|
||
sudo du -sh /mnt/dest/backup-name/
|
||
|
||
# Dry-run to confirm no remaining files
|
||
sudo rsync -avhWn /mnt/source/ /mnt/dest/backup-name/ \
|
||
--exclude='$RECYCLE.BIN' --exclude='System Volume Information'
|
||
```
|
||
|
||
### Step 4: Monitor
|
||
|
||
If running in background, check periodically:
|
||
```bash
|
||
ps aux | grep rsync
|
||
# D state = stuck in disk I/O (bad sign if prolonged)
|
||
# S state = sleeping/waiting (normal)
|
||
# R state = actively reading/writing (good)
|
||
```
|
||
|
||
If stuck in D state for >5 minutes with no progress, the process is likely hung on USB I/O contention. Kill and restart with a single instance.
|
||
|
||
## Post-Backup Reorganization
|
||
|
||
After a backup completes, users often want the data at the root of the destination drive rather than nested in a subdirectory:
|
||
|
||
```bash
|
||
# Before: /mnt/dest/backup-name/Photos/, /mnt/dest/backup-name/Videos/
|
||
# After: /mnt/dest/Photos/, /mnt/dest/Videos/
|
||
|
||
# Move everything out of the subdirectory (same filesystem = instant, no data copy)
|
||
sudo mv /mnt/dest/backup-name/* /mnt/dest/
|
||
sudo mv /mnt/dest/backup-name/.* /mnt/dest/ 2>/dev/null # hidden files
|
||
sudo rmdir /mnt/dest/backup-name/
|
||
|
||
# Optionally clean up junk that snuck through from early rsync runs
|
||
sudo rm -rf "/mnt/dest/Recovered data*" "/mnt/dest/System Volume Information" /mnt/dest/msdia80.dll
|
||
```
|
||
|
||
### Merging recovered data into existing directories
|
||
|
||
Data recovery tools (like those found in `Deep Scan result/` folders) often organize recovered files by camera make/model or file type. Users commonly want these merged into their existing organized library:
|
||
|
||
```bash
|
||
# Before merge:
|
||
# /mnt/dest/Photos/ (existing backup)
|
||
# /mnt/dest/Deep Scan result/Photos_deepscan/Camera/ (recovered photos)
|
||
# /mnt/dest/Deep Scan result/Videos/More Lost Files(RAW)/ (recovered videos)
|
||
|
||
# 1. Separate photo and video folders within the recovered data
|
||
mkdir -p "Deep Scan result/Photos" "Deep Scan result/Videos"
|
||
mv "Deep Scan result/Photos_deepscan/Camera" "Deep Scan result/Photos/"
|
||
mv "Deep Scan result/Videos More Lost Files(RAW)" "Deep Scan result/Videos/"
|
||
mv "Deep Scan result/Videos Mov" "Deep Scan result/Videos/"
|
||
|
||
# 2. Merge into main directories
|
||
mv "Deep Scan result/Photos/Camera" /mnt/dest/Photos/
|
||
mv "Deep Scan result/Videos/More Lost Files(RAW)" /mnt/dest/Videos/
|
||
mv "Deep Scan result/Videos/Mov" /mnt/dest/Videos/
|
||
```
|
||
|
||
This is always instant (same-filesystem moves, no data copying). Only directory metadata is updated.
|
||
|
||
## Performance Expectations
|
||
|
||
File type dramatically affects transfer speed on USB drives:
|
||
|
||
| File Type | Typical Speed | Why |
|
||
|-----------|--------------|-----|
|
||
| Large videos (500MB+) | **75–100 MB/s** | Sequential reads, minimal metadata overhead |
|
||
| Photos (2-10MB) | **40–60 MB/s** | Mixed sequential/random |
|
||
| Small files (<1MB, thumbnails, metadata) | **15–40 MB/s** | Directory creation, metadata overhead, random I/O |
|
||
| Mixed (full drive backup) | **50–75 MB/s avg** | Depends on file size distribution |
|
||
|
||
**Expect the tail to slow down.** After large video files finish, the remaining small files (Immich thumbnails, library metadata) will drop to 15-40 MB/s. A 642 GB backup might take ~1.5-2 hours despite the first 500 GB flying through.
|
||
|
||
## Pitfalls
|
||
|
||
- **D state panic:** One or more rsync processes in D state (uninterruptible sleep) for extended periods usually means USB I/O contention. Kill them all and restart with a single sequential instance.
|
||
- **Permission denied on destination:** First rsync run without sudo creates root-owned directories. Subsequent runs as user fail. Use `sudo` consistently, or `chown` the dest after.
|
||
- **Silent permission-denied pattern (diagnostic):** When rsync produces thousands of progress lines (`to-chk` counts down from N to 0) but transfers 0% with `xfr#0` and exits code 23, the mkdir failed at the very start. The actual error (`recv_generator: mkdir "/dest/dir" failed: Permission denied (13)`) only appears ONCE near the top of the output, buried among 2800+ progress lines. The fix is `sudo chown user:user /mnt/dest/` (or `sudo rsync`). Don't waste time reading pages of progress — grep for `denied` or `error` first.
|
||
- **Space miscalculation:** ext4 reserves 5% of blocks for root (default). On an 8TB drive, that's ~372GB "missing" from what the user expects. Check with `tune2fs -l /dev/sdX1 | grep Reserved` and explain decimal-vs-binary if questioned.
|
||
- **Partial completion:** If rsync exits/crashes (common with USB), the next run with the same flags is incremental — it only copies what's missing. No need to start over.
|
||
- **WATCH OUT for stale temp files:** Chrome `.crdownload` and `part*.tmp` files from interrupted downloads waste space and confuse size estimates. Clean them before estimating disk usage.
|
||
- **SSH quoting of `$` in exclude patterns:** When running rsync via `ssh host 'nohup sudo rsync ... --exclude='\''$RECYCLE.BIN'\'' ...'`, the `$` in `$RECYCLE.BIN` requires careful shell quoting. The `'\''...'\''` pattern (break out of outer single quotes, insert literal `'`, re-enter single quotes) works but is fragile. **Safer alternative:** Write the full command to a script on the remote server first, or use a heredoc-style variable on the remote side. A missed `$` expansion means the exclude silently becomes `--exclude=.BIN` (empty variable), and the junk folder gets copied.
|
||
- **`du` is slow on permission-heavy dirs:** Using `sudo du -sh /mnt/dest/backup/` to check progress is slow (minutes) when the dest has thousands of Immich-style hex-nested directories with mixed Docker permissions. **Prefer `df -h /mnt/dest/` for a fast byte-level snapshot** — it shows used space on the whole filesystem, which is accurate enough for progress monitoring. Reserve `du` for final verification.
|