hermes-config/skills/software-development/immich-server/references/yolo-gpu-classification.md

# YOLO GPU Classification on GTX 1050 Ti (Pascal, CC 6.1)

## CUDA/PyTorch Compatibility Matrix

| PyTorch | CUDA | Python | GTX 1050 Ti Result |
|---------|------|--------|-------------------|
| 2.12.0+cu130 | 13.0 | 3.14 | ❌ `no kernel image is available for execution on the device` |
| 2.12.0+cu130 | 13.0 | 3.14 | ❌ `CUDA error: out of memory` (when zombie process holds VRAM) |
| 2.5.1+cu124 | 12.4 | 3.12 | ✅ Working |

**Root cause:** PyTorch 2.12.0 (CUDA 13.0) dropped support for Pascal GPUs (compute capability 6.1) at the binary level — the shipped PTX/SASS targets newer architectures. The only PyTorch wheels available for Python 3.14 are 2.12.0+cu130, so **Python 3.12 must be used** with an older PyTorch + CUDA 12.4 build.

## Setup (Python 3.12 + CUDA 12.4)

```bash
# Install Python 3.12 via deadsnakes (Ubuntu 26.04 doesn't ship it)
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update
sudo apt install -y python3.12 python3.12-venv

# Create venv with CUDA 12.4 PyTorch
python3.12 -m venv /home/ray/yolo_venv_cu124
source /home/ray/yolo_venv_cu124/bin/activate
pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 --index-url https://download.pytorch.org/whl/cu124
pip install ultralytics

# Verify (should show CUDA: True, CC: (6, 1))
python3 -c 'import torch; print(f"CUDA: {torch.cuda.is_available()}, CC: {torch.cuda.get_device_capability(0)}")'
```

**Why Python 3.12?** Ubuntu 26.04 ships Python 3.14.4 as default. PyTorch 2.5.x has no wheels for Python 3.14 — only 2.12.0+cu130, which doesn't support Pascal GPUs. Python 3.12 is the newest Python with CUDA 12.4 PyTorch wheels available via deadsnakes PPA.

## 🚨 Pitfalls

### Zombie GPU Processes After Background Kill

When Hermes kills a background process (e.g. `process(action="kill")`), the SSH session dies but the Python process on the remote server **keeps running and holds GPU VRAM**. This causes subsequent runs to OOM even with a compatible PyTorch version.

**Always check before launching a new GPU run:**

```bash
# Check for zombies
nvidia-smi
# Look for python3 processes in the "Processes" section

# Kill all yolo processes
pkill -f 'seagate_photos_yolo'   # or your script name
# Wait 1-2 seconds for GPU memory to release
sleep 2
nvidia-smi   # Verify 0 MiB used
```

**Symptom:** `CUDA error: out of memory` on a fresh run even though the script should fit in VRAM.

### Corrupt Image Fallback: Don't Retry on GPU

When a batch of images fails YOLO processing (corrupt JPEGs, truncated files), **do NOT retry each file individually on GPU**. The individual fallback path (`model([filepath], imgsz=384, ...)`) is:
- Slow (model re-inference per file)
- Unreliable (corrupt file still fails at lower resolution)
- Wastes GPU cycles on unreadable data

**Instead:** On batch failure, move the entire batch directly to the target folder without GPU retry:

```python
except Exception as e:
    # Batch-level failure — likely corrupt files
    for filepath in batch:
        try:
            rel_path = os.path.relpath(filepath, SRC)
            dst_path = os.path.join(DST, rel_path)
            os.makedirs(os.path.dirname(dst_path), exist_ok=True)
            shutil.move(filepath, dst_path)
            moved += 1
        except Exception:
            errors += 1
    continue
```

### Background Output Buffering

Hermes' `process(action="poll")` may not show stdout from background SSH sessions due to pipe buffering. **Redirect output to a log file and read it with `read_file`** instead:

```bash
# In background command:
python3 -u /tmp/script.py > /tmp/script.log 2>&1

# Monitor progress:
tail -3 /tmp/script.log
# Or use read_file tool
```

Use `python3 -u` (unbuffered) and `flush=True` on all print() calls.

## Full Working Script (Production)

The production script `yolo_gpu_immediate.py` (deployed at `/tmp/yolo_gpu_immediate.py` on rayserver) combines:

- Immich DB query via `docker exec immich_postgres psql ...`
- Path mapping from container `/data/library/...` to real `/mnt/wd-passport/immich/photos/library/...`
- YOLOv8n single-image GPU inference at ~6-7 img/s on GTX 1050 Ti
- **Immediate `shutil.move()` per image** — not batched at end (user preference)
- **`stream=True`** — prevents Ultralytics RAM accumulation warning
- Progress logging every 500 images

### Querying Immich's PostgreSQL

```sql
SELECT "originalPath" FROM asset
WHERE type='IMAGE' AND visibility='timeline'
  AND "originalPath" LIKE '/data/library/%'
ORDER BY "originalPath";
```

Executed via:
```bash
docker exec immich_postgres psql -U postgres -d immich -t -A -c "<SQL>"
```

The `"originalPath"` from Immich looks like `/data/library/4defcb72-1058-4fcd-826a-e2e87e59aa4c/2026/2026-05-11/...`. To map to real filesystem, strip the `/data/` prefix and prepend the real photo base (`/mnt/wd-passport/immich/photos`).

### Classification Ruleset

- **Person** (class 0) → keep
- **Nature** (classes 16-27=birds/cats/dogs/horses/sheep/cow/elephant/bear/zebra/giraffe, 58=potted plant, 77=vase, 80=umbrella) → keep
- **Urban/indoor move-ables** (classes 1-15 = human artifacts like car/bicycle/motorcycle/airplane/bus/train/truck/boat/traffic light/fire hydrant/stop sign/parking meter/bench, 56=chair, 57=couch, 59=bed, 60=dining table, 61=toilet, 62=tv, 63=laptop, 64=mouse, 65=remote, 67=cell phone, 70=oven, 71=toaster, 72=sink, 73=refrigerator, 74=book, 75=clock, 76=keyboard, 78=scissors, 79=teddy bear) → move
- **No detections + file <400KB** → move (screenshots, memes)
- **No detections + file >=400KB** → keep (likely scenic with no recognizable objects)

### GPU Memory Profile

During single-image inference on GTX 1050 Ti (4GB):
- Memory used: ~445 MB
- GPU utilization: ~5% (bottlenecked by image loading/preprocessing, not compute)
- Batch size 4 can increase utilization but risks OOM for larger images

## Background Execution (Keep Agent Responsive)

The user explicitly wants all long-running tasks in background so Hermes stays accessible.

**Pattern: redirect to log file, monitor with read_file.**

```bash
# Start (via Hermes terminal background=true)
ssh rayserver "source /home/ray/yolo_venv_cu124/bin/activate && python3 -u /tmp/script.py > /tmp/yolo_seagate.log 2>&1"

# Monitor progress — use read_file, NOT process poll (buffering issue)
read_file /tmp/yolo_seagate.log

# Files moved so far
find /mnt/seagate8tb/NO\ PEOPLE/ -type f | wc -l

# Kill zombies (check nvidia-smi first!)
pkill -f script_name
nvidia-smi   # verify GPU memory freed
```

**Never use `tail` via terminal for log monitoring** — use `read_file` instead. Background SSH stdout is buffered at the pipe level and `process(action="poll")` often shows empty output even when the script is actively writing.

## EXIF Rotation: Critical Fix for Phone Photos

**YOLO via OpenCV ignores EXIF orientation.** Phone portrait photos (which are stored landscape with an EXIF rotation flag) will appear sideways to the model. People in these photos are routinely missed.

### Fix: PIL load with EXIF transpose

```python
from PIL import Image, ImageOps
import numpy as np

with Image.open(ap) as img:
    img = ImageOps.exif_transpose(img)
    if img is None:
        img = Image.open(ap)  # reload if exif_transpose returned None
    if img.mode != 'RGB':
        img = img.convert('RGB')  # handle RGBA (PNG alpha channel), CMYK, etc.
    img_np = np.array(img)[:, :, ::-1]  # RGB -> BGR for Ultralytics pipeline
    results = model(img_np, device="cuda:0", verbose=False)
```

**Important:** After `exif_transpose`, the image MUST be converted to `'RGB'` mode. PNGs with alpha channels are `RGBA` (4 channels) and YOLO expects exactly 3.

### 4-Orientation Brute-Force (When EXIF Alone Fails)

Some photos are stored rotated in pixel data WITHOUT any EXIF orientation flag. `ImageOps.exif_transpose()` returns them unchanged. Fix: try all 4 rotations:

```python
import numpy as np

# After PIL load (RGB), convert to BGR for YOLO pipeline
img_bgr = np.array(img)[:, :, ::-1]

orientations = {
    '0°': img_bgr,
    '90°': np.rot90(img_bgr, k=3),   # 90° CW
    '180°': np.rot90(img_bgr, k=2),  # 180°
    '270°': np.rot90(img_bgr, k=1),  # 270° CW
}

found_person = False
for orient_name, orient_img in orientations.items():
    results = model(orient_img, device="cuda:0", verbose=False)
    for r in results:
        if r.boxes:
            for box in r.boxes:
                if int(box.cls[0]) == 0:  # person class
                    found_person = True
                    break
        if found_person: break
    if found_person: break
```

**Performance impact:** 4x slower (~2.5-3 effective img/s on GTX 1050 Ti). Run as background task. Typically restores 2-7% of flagged photos as having people.

### PIL Decompression Bomb Protection

Pillow refuses to load images larger than 178MP by default (decompression bomb safety). Photos from phone panoramas or photo stitches (200MP+) will fail with:
```
Image size (199756800 pixels) exceeds limit of 178956970 pixels
```

These images cannot be checked via PIL. Two options:
- **Skip them** (they stay in their current classification, usually with a count in the error tally)
- **Raise the limit** with `Image.MAX_IMAGE_PIXELS = None` (not recommended — can cause OOM on 4GB GPU)

## Post-Classification Sorting: Trash vs Keep

After moving non-people photos to a separate folder, use YOLO's object detections to categorize them into Trash (likely insignificant) and Keep (potentially important).

### Classification Rules

**TRASH** (screenshots, household clutter, no meaningful content):
- cell phone (67), tv (62), laptop (63), keyboard (66), mouse (64), remote (65)
- toilet (61), chair (56), couch (57), refrigerator (72), microwave (68), oven (69), sink (71)
- hair drier (78), toothbrush (79), scissors (76)
- No detections at all (blank/solid/text-only images)

**KEEP** (potentially important):
- book (73) — documents, receipts, textbooks
- car (2), truck (7), bus (5), motorcycle (3), bicycle (1), airplane (4) — vehicles/travel
- dining table (60), cup (41), bottle (39), bowl (45), wine glass (40) — food/meals
- dog (16), cat (15), bird (14) — pets
- sports ball (32), cake (55), pizza (53) — events/food
- clock (74) — time-stamped events
- potted plant (58), vase (75), backpack (24), umbrella (25) — personal/meaningful

A photo with ANY "Keep" class detected goes to Keep. ONLY photos where ALL detected classes are Trash go to Trash. No-detection photos go to Trash.

### Typical Distribution

On a real-world photo library (16K photos), after YOLOv8n classification:
- **~24%** flagged for removal (no people, no scenic content)
- **~22%** of that flagged set is screenshots/trash
- **~55%** is documents/books
- **~15%** is vehicles/meals/personal worth keeping
- **~3%** has missed people in rotated orientation (restored via multi-orientation check)

## Restore Pass: Re-Checking Flagged Photos

After the initial classification pass, ALWAYS do a restore pass on the moved photos. The user will spot false positives. The restore workflow:

1. Scan all files in the NO PEOPLE folder
2. For each, load with PIL + EXIF transpose + 4 orientations
3. If ANY orientation detects a person (class 0), `shutil.move()` back to original path
4. Log which orientation caught it (helps confirm the fix works)

```python
# Restore path calculation
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
os.makedirs(os.path.dirname(orig), exist_ok=True)
shutil.move(ap, orig)
```

## Flattening Moved Photos

After moving flagged photos out of Immich's structured directory tree, flatten them into a single folder for easier browsing:

```bash
mkdir -p "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type f -not -path '*/ALL Photos/*' -exec mv -t "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/" {} +
```

Handle filename collisions with number suffixes.

## Full Pipeline Workflow Order

The complete YOLO classification workflow runs in this order:

```
1. CLASSIFY  → 2. RESTORE  → 3. FLATTEN  → 4. SORT
   (initial)    (re-check)    (remove      (trash vs keep
                4-orient)     subdirs)     by detection)
```

### Step 1: Classify
- Query Immich DB for all image paths
- Map container paths to real filesystem paths
- Run YOLOv8n on GPU, immediate `shutil.move()` on non-people/non-scenic photos
- Log progress every 500 images

### Step 2: Restore (Multi-Orientation Re-check)
- Scan all moved files in the NO PEOPLE target folder
- For each, load with PIL + EXIF transpose + try all 4 orientations
- If ANY orientation detects a person (class 0), move back to original path
- Log which orientation caught the detection
- Typically restores 2-7% of flagged photos

### Step 3: Flatten
- After restore pass, gather all remaining photos into a single flat directory
- Remove the deep Immich subdirectory structure (UUID/YYYY/YYYY-MM-DD/)
- Handle filename collisions with `_1`, `_2` suffixes
- Result: one folder with N files, no subdirectories

### Step 4: Sort (Trash vs Keep)
- Run YOLO on all flattened photos to detect what objects are present
- Categorize each photo:
  - **Trash**: ALL detected classes are screenshots/clutter AND no meaningful detections
  - **Keep**: ANY detected class is a "keep" class (vehicles, food, pets, documents, etc.)
- Move into Trash/ and Keep/ subfolders
- Photos with no detections → Trash

### Flatten Script (with Collision Handling)

```python
import os, shutil

SRC = "/path/to/NO PEOPLE PHOTOS"
DST = os.path.join(SRC, "ALL Photos")
os.makedirs(DST, exist_ok=True)

files = []
for root, dirs, fnames in os.walk(SRC):
    if root.startswith(DST):
        continue
    for f in fnames:
        files.append(os.path.join(root, f))

collisions = 0
for src in files:
    fname = os.path.basename(src)
    dst = os.path.join(DST, fname)
    if os.path.exists(dst):
        base, ext = os.path.splitext(fname)
        n = 1
        while os.path.exists(os.path.join(DST, f"{base}_{n}{ext}")):
            n += 1
        dst = os.path.join(DST, f"{base}_{n}{ext}")
        collisions += 1
    shutil.move(src, dst)

print(f"Moved {len(files)} files, {collisions} renamed")
```

### Sort Script (Trash vs Keep)

```python
import os, shutil, csv

# See reference above for TRASH_CLASSES set definition
# See the Post-Classification Sorting section above for the detailed class lists
# CSV file produced by the categorization run: filename,detected_classes
# A photo goes to KEEP if ANY detected class is NOT in the TRASH set
# Otherwise it goes to TRASH (which also catches no-detection photos)
```

## Full Pipeline Script Template

For a complete end-to-end script combining classification, EXIF rotation, 4-orientation restore, and sorting, combine the patterns above. Run as background process with nohup. The user expects to check progress periodically and be able to interact with the agent while the job runs.