hermes-config/skills/devops/storage-management/SKILL.md

name, description, version, author, license, platforms, metadata, related_skills

name	description	version	author	license	platforms	metadata	related_skills
storage-management	Detect, partition, format, mount, and back up storage drives on Linux servers — including fstab setup, rsync backup workflows, and permission management.	1.0.0	Hermes Agent	MIT	linux	hermes tags storage disks drives rsync backup fstab mount	server-health-check

Storage & Drive Management

Manage physical storage drives on Linux — detection, partitioning, formatting, persistent mounting, and full-drive backup via rsync.

Detecting New Drives

# Full overview: include TRAN(transport) to distinguish USB from SATA/NVMe
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN

# Unmounted drives (device listed but no mountpoint)
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,TRAN | grep -E 'disk$|part$' | awk '$5 == ""'

# Get UUIDs for fstab
sudo blkid

# Find by USB vendor/model (useful for identifying thumb drives)
ls -la /dev/disk/by-id/usb* 2>/dev/null

The TRAN column shows usb, sata, or nvme — immediately tells you how the drive is connected. A drive showing in lsblk with no partitions and no mount point is ready to be set up.

Partitioning & Formatting

Step 1 — Create GPT partition table + single partition

sudo parted /dev/sdX mklabel gpt
sudo parted /dev/sdX mkpart primary ext4 0% 100%
sudo partprobe /dev/sdX    # reload partition table

Step 2 — Format with ext4 (label it for easy identification)

sudo mkfs.ext4 /dev/sdX1 -L DriveLabel

Note: mkfs is on the unconditional blocklist in Hermes. The agent cannot run it. Ask the user to run the command directly, then proceed with mounting and fstab.

Step 3 — Create mount point and mount

sudo mkdir -p /mnt/drive-label
sudo mount /dev/sdX1 /mnt/drive-label

Step 4 — Add to fstab for persistence

# Get UUID
sudo blkid /dev/sdX1

# Add entry (ext4 example)
echo 'UUID=xxxx-xxxx /mnt/drive-label ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab

fstab format: <file_system> <mount_point> <type> <options> <dump> <pass>

Field	Meaning
UUID=...	Use UUID, not /dev/sdX — survives re-enumeration
mount point	Directory to mount at
type	ext4, ntfs-3g, exfat, vfat
options	defaults, noatime (skip access time updates), uid=1000,gid=1000,umask=000 (permissions for NTFS/exFAT)
dump	0 (no backup)
pass	2 (non-root fsck), 1 (root fsck), 0 (no fsck)

External/Removable Drives

ALWAYS add nofail to external drives in fstab. Without it, systemd waits for the drive at boot and either hangs or drops to emergency mode if the drive is unplugged.

# WRONG — hangs at boot if drive missing
UUID=xxxx /mnt/external ext4 defaults,noatime 0 2

# RIGHT — skips silently if drive not present
UUID=xxxx /mnt/external ext4 defaults,noatime,nofail 0 0

Also use pass=0 (last column) for external drives — a missing drive that's flagged for fsck (0 2) will cause a boot stall even with nofail.

Step 5 — Verify

sudo mount -a    # re-read fstab, mount any missing entries
df -h /mnt/drive-label
lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT

Unmounting Drives

Always unmount before physically disconnecting a drive to avoid data corruption.

sudo umount /mnt/drive-label

When umount hangs or times out

If umount hangs (e.g., a process still has open file handles or I/O is stalled), use lazy unmount:

sudo umount -l /mnt/drive-label

-l (lazy) detaches the filesystem from the mount tree immediately and cleans up references once they're no longer in use. The drive is safe to disconnect after umount -l returns successfully — you'll either get a clean detach or not mounted (meaning the original umount actually completed despite the timeout).

If even umount -l reports not mounted, the drive is already detached — safe to remove.

Filesystem-Specific Notes

FS	Install Package	Mount Type	Special Options
ext4	built-in	default	defaults,noatime
NTFS	ntfs-3g	ntfs-3g	uid=1000,gid=1000,umask=000 (full user access)
exFAT	exfatprogs	exfat	uid=1000,gid=1000,umask=000

⚠️ NTFS on Linux: Always specify -t ntfs-3g explicitly. The kernel ntfs3 driver (in-tree) is experimental and can corrupt data. Use the userspace ntfs-3g which is battle-tested.

Full-Drive Backup with rsync

For copying an entire drive (or large directories) to another local drive:

Single-Instance rsync (Recommended for USB Drives)

Multiple parallel rsync instances on the same USB source cause I/O contention errors (error in socket IO, Broken pipe, errors selecting input/output files). Always use a single instance when the source is a USB/external drive.

# Local disk-to-disk copy (fastest approach)
sudo rsync -avhW --progress /mnt/source/ /mnt/dest/backup-folder/ \
  --exclude='$RECYCLE.BIN' --exclude='System Volume Information'

Flag	Meaning
-a	Archive mode (preserves permissions, timestamps, symlinks)
-v	Verbose
-h	Human-readable sizes
-W	Whole-file (skip delta checksum) — faster for local copies
--progress	Show per-file transfer progress

Why -W for local copies: rsync normally reads blocks of each file to compute checksums for delta transfer. On a local copy there's no network benefit — -W just copies the whole file, eliminating the checksum overhead.

For deep rsync best practices (USB I/O contention, ionice scheduling, progress monitoring, post-backup reorganization, performance expectations), see references/rsync-best-practices.md (absorbed from data-migration).

Permission Handling

• Before running non-sudo rsync, check destination ownership: ls -la /mnt/dest/ — if the parent directory is root-owned (e.g., drwxr-xr-x root root), rsync will fail with Permission denied on mkdir. Fix: sudo chown user:user /mnt/dest/ first.
• If rsync runs with sudo, dest dirs are owned by root
• Subsequent non-sudo rsync runs fail with Permission denied on mkdir
• Fix: sudo chown -R user:user /mnt/dest/ or chmod on target dirs
• Or just always run the rsync with sudo

Progress Monitoring

# Check size transferred so far
df -h /mnt/dest-drive

# Check process is alive
ps aux | grep rsync | grep -v grep

# Recent log lines
tail -5 /path/to/rsync.log

# File count (slow for large dirs)
find /mnt/dest/backup-folder/ -type f | wc -l

Expected Speeds

Source Type	Typical Speed	Notes
USB 3.0 HDD → SATA SSD	80-120 MB/s	Source-limited
USB 2.0 HDD → any	20-40 MB/s	Interface bottleneck
NVMe → NVMe	500+ MB/s	Only if both are internal

Resume Behavior

rsync automatically resumes partial transfers. Files with matching name, size, and mtime are skipped. You can safely kill and restart — only new/changed files transfer.

Exposing Drives Over the Network (Samba)

After mounting a drive, you can share it over the network via Samba (SMB/CIFS) so Windows, macOS, and Linux machines on the LAN can access it as a network drive.

Check Samba Status

which smbd && smbd --version   # check if installed
cat /etc/samba/smb.conf        # existing config

Add a New Share

Samba uses a simple INI-style config at /etc/samba/smb.conf. Append a new share section:

sudo tee -a /etc/samba/smb.conf << 'EOF'

[share-name]
path = /mnt/drive
browseable = yes
read only = no
guest ok = yes
force user = ray
create mask = 0777
directory mask = 0777
EOF

Option	Meaning
guest ok = yes	No password required on connect
force user = ray	All files written over SMB owned by ray
create mask = 0777	New files get full permissions
directory mask = 0777	New directories get full permissions

Restart & Verify

sudo systemctl restart smbd

# Verify the share shows up
smbclient -L //localhost -U ray --no-pass | grep share-name

# Find server LAN IP
ip -4 addr show | grep -oP 'inet \K[^/]+' | grep -v 127.0.0.1

Connecting from Clients

OS	How
Windows	\\192.168.x.x\share-name in File Explorer
macOS	⌘K → smb://192.168.x.x/share-name
Linux	smb://192.168.x.x/share-name in file manager

Password-Protected Shares (Alternative)

sudo smbpasswd -a ray                    # set SMB password (interactive)
# or from a script:
echo 'password' | sudo smbpasswd -a -s ray

Then use valid users = ray and guest ok = no in the share section.

Pitfalls

• SMB password ≠ system password. Setting a Samba password with smbpasswd is a separate step from the user's Linux login password.
• $ in share paths: The $RECYCLE.BIN directory on NTFS drives contains a literal $. When writing Samba configs or rsync excludes, the $ must be quoted or escaped to prevent shell variable expansion.
• Firewall: If clients can't connect, check Samba ports: sudo ufw allow samba or sudo ufw allow 139,445/tcp.

Pitfalls

• Mounted drives are root-owned by default — cp, rsync, mkdir, and any write to a freshly mounted drive will fail with Permission denied unless you use sudo. After sudo operations, subdirectories are owned by root and non-sudo writes keep failing. Either always use sudo for writes to mounted drives, or sudo chown user:user /mnt/mountpoint/ after mounting to grant user-level access.
• fstab doesn't evaluate shell commands — UUID=$(sudo blkid ...) doesn't work in fstab. Use the literal UUID from sudo blkid.
• External drives MUST have nofail — missing external drives without nofail cause systemd to hang at boot (waits 90s per drive) or drop to emergency mode. Always add nofail and pass=0 to fstab entries for removable/external drives.
• Parallel rsync on USB drives causes I/O error / Broken pipe / socket IO error. The USB controller can't handle concurrent reads. Use a single instance.
• Drive re-enumeration — /dev/sdb may become /dev/sdc after reboot or replug. Always use UUIDs in fstab.
• Permissions gap after sudo rsync — subsequent non-sudo runs fail on directories the first run created as root. Either always use sudo, or chown after the first run.
• NTFS write support — ntfs-3g not ntfs3. The kernel ntfs3 driver is experimental. Install ntfs-3g explicitly.
• exFAT on modern Ubuntu — package is exfatprogs (not exfat-utils which is deprecated). Ubuntu 24.04+ has kernel exFAT support, but still needs the tools package.

References

• references/drive-selection-guide.md — SMR vs CMR guidance, speed tiers, physical/electrical compatibility checklist, OEM RAM upgrade assessment (for when the user asks "will this drive fit?" or "how does this compare to my current drive?")
• references/rsync-best-practices.md — rsync pattern reference for USB I/O
• references/rayserver-shares.md — rayserver-specific Samba share inventory (absorbed from samba-nas)