hermes-config/skills/devops/storage-management/SKILL.md

---
name: storage-management
description: Detect, partition, format, mount, and back up storage drives on Linux servers — including fstab setup, rsync backup workflows, and permission management.
version: 1.0.0
author: Hermes Agent
license: MIT
platforms: [linux]
metadata:
  hermes:
    tags: [storage, disks, drives, rsync, backup, fstab, mount]
related_skills: [server-health-check]
---

# Storage & Drive Management

Manage physical storage drives on Linux — detection, partitioning, formatting, persistent mounting, and full-drive backup via rsync.

## Detecting New Drives

```bash
# Full overview: include TRAN(transport) to distinguish USB from SATA/NVMe
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN

# Unmounted drives (device listed but no mountpoint)
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN | grep -E 'disk$|part$' | awk '$5 == ""'

# Get UUIDs for fstab
sudo blkid

# Find by USB vendor/model (useful for identifying thumb drives)
ls -la /dev/disk/by-id/usb* 2>/dev/null
```

The `TRAN` column shows `usb`, `sata`, or `nvme` — immediately tells you how the drive is connected.
A drive showing in `lsblk` with no partitions and no mount point is ready to be set up.

## Partitioning & Formatting

### Step 1 — Create GPT partition table + single partition

```bash
sudo parted /dev/sdX mklabel gpt
sudo parted /dev/sdX mkpart primary ext4 0% 100%
sudo partprobe /dev/sdX    # reload partition table
```

### Step 2 — Format with ext4 (label it for easy identification)

```bash
sudo mkfs.ext4 /dev/sdX1 -L DriveLabel
```

**Note:** `mkfs` is on the unconditional blocklist in Hermes. The agent cannot run it. Ask the user to run the command directly, then proceed with mounting and fstab.

### Step 3 — Create mount point and mount

```bash
sudo mkdir -p /mnt/drive-label
sudo mount /dev/sdX1 /mnt/drive-label
```

### Step 4 — Add to fstab for persistence

```bash
# Get UUID
sudo blkid /dev/sdX1

# Add entry (ext4 example)
echo 'UUID=xxxx-xxxx /mnt/drive-label ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab
```

**fstab format:** `<file_system> <mount_point> <type> <options> <dump> <pass>`

| Field | Meaning |
|-------|---------|
| `UUID=...` | Use UUID, not `/dev/sdX` — survives re-enumeration |
| mount point | Directory to mount at |
| type | `ext4`, `ntfs-3g`, `exfat`, `vfat` |
| options | `defaults`, `noatime` (skip access time updates), `uid=1000,gid=1000,umask=000` (permissions for NTFS/exFAT) |
| dump | 0 (no backup) |
| pass | `2` (non-root fsck), `1` (root fsck), `0` (no fsck) |

### External/Removable Drives

**ALWAYS add `nofail` to external drives in fstab.** Without it, systemd waits for the drive at boot and either hangs or drops to emergency mode if the drive is unplugged.

```bash
# WRONG — hangs at boot if drive missing
UUID=xxxx /mnt/external ext4 defaults,noatime 0 2

# RIGHT — skips silently if drive not present
UUID=xxxx /mnt/external ext4 defaults,noatime,nofail 0 0
```

Also use `pass=0` (last column) for external drives — a missing drive that's flagged for fsck (`0 2`) will cause a boot stall even with `nofail`.

### Step 5 — Verify

```bash
sudo mount -a    # re-read fstab, mount any missing entries
df -h /mnt/drive-label
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT
```

## Unmounting Drives

Always unmount before physically disconnecting a drive to avoid data corruption.

```bash
sudo umount /mnt/drive-label
```

### When umount hangs or times out

If `umount` hangs (e.g., a process still has open file handles or I/O is stalled), use lazy unmount:

```bash
sudo umount -l /mnt/drive-label
```

`-l` (lazy) detaches the filesystem from the mount tree immediately and cleans up references once they're no longer in use. The drive is safe to disconnect after `umount -l` returns successfully — you'll either get a clean detach or `not mounted` (meaning the original umount actually completed despite the timeout).

If even `umount -l` reports `not mounted`, the drive is already detached — safe to remove.

### Filesystem-Specific Notes

| FS | Install Package | Mount Type | Special Options |
|----|----------------|------------|-----------------|
| **ext4** | built-in | default | `defaults,noatime` |
| **NTFS** | `ntfs-3g` | `ntfs-3g` | `uid=1000,gid=1000,umask=000` (full user access) |
| **exFAT** | `exfatprogs` | `exfat` | `uid=1000,gid=1000,umask=000` |

> ⚠️ **NTFS on Linux:** Always specify `-t ntfs-3g` explicitly. The kernel `ntfs3` driver (in-tree) is experimental and can corrupt data. Use the userspace `ntfs-3g` which is battle-tested.

## Full-Drive Backup with rsync

For copying an entire drive (or large directories) to another local drive:

### Single-Instance rsync (Recommended for USB Drives)

Multiple parallel rsync instances on the same USB source cause I/O contention errors (`error in socket IO`, `Broken pipe`, `errors selecting input/output files`). Always use a **single instance** when the source is a USB/external drive.

```bash
# Local disk-to-disk copy (fastest approach)
sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-folder/ \
  --exclude='$RECYCLE.BIN' --exclude='System Volume Information'
```

| Flag | Meaning |
|------|---------|
| `-a` | Archive mode (preserves permissions, timestamps, symlinks) |
| `-v` | Verbose |
| `-h` | Human-readable sizes |
| `-W` | **Whole-file** (skip delta checksum) — faster for local copies |
| `--progress` | Show per-file transfer progress |

**Why `-W` for local copies:** rsync normally reads blocks of each file to compute checksums for delta transfer. On a local copy there's no network benefit — `-W` just copies the whole file, eliminating the checksum overhead.

For deep rsync best practices (USB I/O contention, ionice scheduling, progress monitoring, post-backup reorganization, performance expectations), see `references/rsync-best-practices.md` (absorbed from `data-migration`).

### Permission Handling

- **Before running non-sudo rsync, check destination ownership:** `ls -la /mnt/dest/` — if the parent directory is root-owned (e.g., `drwxr-xr-x root root`), rsync will fail with `Permission denied` on `mkdir`. Fix: `sudo chown user:user /mnt/dest/` first.
- If rsync runs with `sudo`, dest dirs are owned by root
- Subsequent non-sudo rsync runs fail with `Permission denied` on mkdir
- **Fix:** `sudo chown -R user:user /mnt/dest/` or chmod on target dirs
- Or just always run the rsync with `sudo`

### Progress Monitoring

```bash
# Check size transferred so far
df -h /mnt/dest-drive

# Check process is alive
ps aux | grep rsync | grep -v grep

# Recent log lines
tail -5 /path/to/rsync.log

# File count (slow for large dirs)
find /mnt/dest/backup-folder/ -type f | wc -l
```

### Expected Speeds

| Source Type | Typical Speed | Notes |
|-------------|--------------|-------|
| USB 3.0 HDD → SATA SSD | 80-120 MB/s | Source-limited |
| USB 2.0 HDD → any | 20-40 MB/s | Interface bottleneck |
| NVMe → NVMe | 500+ MB/s | Only if both are internal |

### Resume Behavior

rsync automatically resumes partial transfers. Files with matching name, size, and mtime are skipped. You can safely kill and restart — only new/changed files transfer.

## Exposing Drives Over the Network (Samba)

After mounting a drive, you can share it over the network via Samba (SMB/CIFS) so Windows, macOS, and Linux machines on the LAN can access it as a network drive.

### Check Samba Status

```bash
which smbd && smbd --version   # check if installed
cat /etc/samba/smb.conf        # existing config
```

### Add a New Share

Samba uses a simple INI-style config at `/etc/samba/smb.conf`. Append a new share section:

```bash
sudo tee -a /etc/samba/smb.conf << 'EOF'

[share-name]
path = /mnt/drive
browseable = yes
read only = no
guest ok = yes
force user = ray
create mask = 0777
directory mask = 0777
EOF
```

| Option | Meaning |
|--------|---------|
| `guest ok = yes` | No password required on connect |
| `force user = ray` | All files written over SMB owned by `ray` |
| `create mask = 0777` | New files get full permissions |
| `directory mask = 0777` | New directories get full permissions |

### Restart & Verify

```bash
sudo systemctl restart smbd

# Verify the share shows up
smbclient -L //localhost -U ray --no-pass | grep share-name

# Find server LAN IP
ip -4 addr show | grep -oP 'inet \K[^/]+' | grep -v 127.0.0.1
```

### Connecting from Clients

| OS | How |
|----|-----|
| **Windows** | `\\192.168.x.x\share-name` in File Explorer |
| **macOS** | ⌘K → `smb://192.168.x.x/share-name` |
| **Linux** | `smb://192.168.x.x/share-name` in file manager |

### Password-Protected Shares (Alternative)

```bash
sudo smbpasswd -a ray                    # set SMB password (interactive)
# or from a script:
echo 'password' | sudo smbpasswd -a -s ray
```

Then use `valid users = ray` and `guest ok = no` in the share section.

### Pitfalls

- **SMB password ≠ system password.** Setting a Samba password with `smbpasswd` is a separate step from the user's Linux login password.
- **`$` in share paths:** The `$RECYCLE.BIN` directory on NTFS drives contains a literal `$`. When writing Samba configs or rsync excludes, the `$` must be quoted or escaped to prevent shell variable expansion.
- **Firewall:** If clients can't connect, check Samba ports: `sudo ufw allow samba` or `sudo ufw allow 139,445/tcp`.

## Pitfalls

- **Mounted drives are root-owned by default** — `cp`, `rsync`, `mkdir`, and any write to a freshly mounted drive will fail with `Permission denied` unless you use `sudo`. After sudo operations, subdirectories are owned by root and non-sudo writes keep failing. Either always use `sudo` for writes to mounted drives, or `sudo chown user:user /mnt/mountpoint/` after mounting to grant user-level access.
- **fstab doesn't evaluate shell commands** — `UUID=$(sudo blkid ...)` doesn't work in fstab. Use the literal UUID from `sudo blkid`.
- **External drives MUST have `nofail`** — missing external drives without `nofail` cause systemd to hang at boot (waits 90s per drive) or drop to emergency mode. Always add `nofail` and `pass=0` to fstab entries for removable/external drives.
- **Parallel rsync on USB drives** causes `I/O error` / `Broken pipe` / `socket IO error`. The USB controller can't handle concurrent reads. Use a single instance.
- **Drive re-enumeration** — `/dev/sdb` may become `/dev/sdc` after reboot or replug. Always use UUIDs in fstab.
- **Permissions gap after sudo rsync** — subsequent non-sudo runs fail on directories the first run created as root. Either always use sudo, or chown after the first run.
- **NTFS write support** — `ntfs-3g` not `ntfs3`. The kernel ntfs3 driver is experimental. Install `ntfs-3g` explicitly.
- **exFAT on modern Ubuntu** — package is `exfatprogs` (not `exfat-utils` which is deprecated). Ubuntu 24.04+ has kernel exFAT support, but still needs the tools package.

## References

- `references/drive-selection-guide.md` — SMR vs CMR guidance, speed tiers, physical/electrical compatibility checklist, OEM RAM upgrade assessment (for when the user asks "will this drive fit?" or "how does this compare to my current drive?")
- `references/rsync-best-practices.md` — rsync pattern reference for USB I/O
- `references/rayserver-shares.md` — rayserver-specific Samba share inventory (absorbed from `samba-nas`)