ray
244 lines
12 KiB
Markdown
244 lines
12 KiB
Markdown
# Immich YOLO Photo Classifier — Production Script
|
||
|
||
A working sequential YOLO classification pipeline for Immich photos on a self-hosted server with GTX 1050 Ti.
|
||
|
||
## Script
|
||
|
||
Saved at `/tmp/yolo_gpu_run.py` on rayserver. Key design:
|
||
|
||
### Workflow
|
||
1. **Query Immich DB** via `docker exec immich_postgres psql` — pulls all timeline IMAGE assets
|
||
2. **Map virtual paths** — `/data/library/...` → `/mnt/wd-passport/immich/photos/...`
|
||
3. **Classify with YOLOv8n** on CUDA — processes one image at a time, logs every 500
|
||
4. **Move non-people, non-scenic photos** to `/mnt/wd-passport/immich/NO PEOPLE PHOTOS`
|
||
|
||
### Classification heuristic (COCO classes)
|
||
```
|
||
PERSON_CLASS = 0 → keep
|
||
NATURE_CLASSES = {16,17,18,19,20,21,22,23,24, → keep
|
||
25,26,27,58,77,80}
|
||
MOVE_CLASSES = {56,57,59–65,67,70–80,1–15} → move (indoor/urban/vehicle/animals)
|
||
No detections + file < 400KB → move
|
||
No detections + file ≥ 400KB → keep as scenic
|
||
```
|
||
|
||
### Running it
|
||
Script is launched with `nohup` so it survives terminal closure:
|
||
```bash
|
||
nohup ~/yolo_venv_cu126/bin/python3 /tmp/yolo_gpu_run.py > ~/yolo_run.log 2>&1
|
||
```
|
||
|
||
Check progress: `tail -5 ~/yolo_run.log`
|
||
|
||
### Performance (GTX 1050 Ti, USB HDD)
|
||
- **16,187 images** in ~40 min (6–7 img/s)
|
||
- **1,875 photos** moved in a test run (12%)
|
||
- Bottleneck is disk I/O (WD Passport USB HDD), not GPU
|
||
|
||
### EXIF-Aware Restore (fixes sideways missed detections — NOT always sufficient)
|
||
|
||
When YOLO misses people in rotated/portrait photos, first try PIL-based EXIF rotation. **But note: EXIF-only is NOT always sufficient.** Some photos are stored with rotated pixel data and no EXIF orientation flag at all. The brute-force 4-orientation approach below covers all cases.
|
||
|
||
#### Approach A: EXIF rotation (fast, catches flagged rotations)
|
||
|
||
```python
|
||
from PIL import Image, ImageOps
|
||
import numpy as np
|
||
from ultralytics import YOLO
|
||
|
||
model = YOLO("yolov8n.pt")
|
||
NO_PEOPLE_DIR = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
|
||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||
|
||
for ap in files:
|
||
# Load with EXIF rotation applied
|
||
with Image.open(ap) as img:
|
||
img = ImageOps.exif_transpose(img)
|
||
if img is None:
|
||
continue
|
||
if img.mode != "RGB":
|
||
img = img.convert("RGB")
|
||
img_np = np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||
|
||
results = model(img_np, device="cuda:0", verbose=False)
|
||
|
||
has_person = any(
|
||
int(box.cls[0]) == 0
|
||
for r in results if r.boxes
|
||
for box in r.boxes
|
||
)
|
||
|
||
if has_person:
|
||
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
|
||
os.makedirs(os.path.dirname(orig), exist_ok=True)
|
||
shutil.move(ap, orig)
|
||
```
|
||
|
||
**Result from 3951 photos**: 0 restores — EXIF rotation found no additional people.
|
||
|
||
#### ✅ Approach B: 4-Orientation brute-force (definitive, catches all rotation cases)
|
||
|
||
When EXIF-only finds nothing but the user insists people exist in sideways photos, switch to trying all 4 orientations. This is the **definitive** approach:
|
||
|
||
```python
|
||
import numpy as np
|
||
|
||
def try_all_orientations(ap: str, model) -> tuple[bool, str]:
|
||
with Image.open(ap) as img:
|
||
if img.mode != 'RGB':
|
||
img = img.convert('RGB')
|
||
img_np = np.array(img)[:, :, ::-1]
|
||
|
||
orientations = {
|
||
'0°': img_np,
|
||
'90°': np.rot90(img_np, k=3),
|
||
'180°': np.rot90(img_np, k=2),
|
||
'270°': np.rot90(img_np, k=1),
|
||
}
|
||
|
||
for orient_name, orient_img in orientations.items():
|
||
results = model(orient_img, device="cuda:0", verbose=False)
|
||
for r in results:
|
||
if r.boxes:
|
||
for box in r.boxes:
|
||
if int(box.cls[0]) == 0:
|
||
return (True, orient_name)
|
||
return (False, "")
|
||
|
||
# Restore loop
|
||
restored = 0
|
||
for ap in files:
|
||
found, orientation = try_all_orientations(ap, model)
|
||
if found:
|
||
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
|
||
os.makedirs(os.path.dirname(orig), exist_ok=True)
|
||
shutil.move(ap, orig)
|
||
restored += 1
|
||
print(f"RESTORED ({orientation}): {os.path.basename(ap)}")
|
||
```
|
||
|
||
**Result from 3951 photos**: **280 restored** (primarily at 90° and 270°). Most missing people were in photos taken on phones that store portrait pixel data without EXIF orientation flags.
|
||
|
||
**Performance**: ~4.5 img/s, ~25 min for 4k photos, RAM stable at 14%.
|
||
|
||
## Image Loading Edge Cases (PIL path)
|
||
|
||
When passing a numpy array (from PIL) instead of a file path to YOLO, these issues arise:
|
||
|
||
- **`ImageOps.exif_transpose` returns None** — This happens when the image has no EXIF orientation tag at all (common with screenshots, downloaded images, re-saved files). The old pattern `if img is None: continue` **silently skips these images** without counting them as errors or "still empty". Fix: reload the image fresh without EXIF if None, or fall through to 4-orientation brute force.
|
||
- **RGBA PNG** (4 channels): `RuntimeError: expected input[1, 4, 640, 416] to have 3 channels, but got 4 channels instead`. Fix: `img = img.convert("RGB")`.
|
||
- **Decompression bomb**: `Image size (199756800 pixels) exceeds limit of 178956970 pixels`. Large panoramas/stitches (~200MP) trigger Pillow's safety limit. Fix: `PIL.Image.MAX_IMAGE_PIXELS = None` before loading, or skip large files.
|
||
- **Corrupt JPEG**: `Invalid SOS parameters for sequential JPEG` / `Corrupt JPEG data: 1 extraneous bytes before marker`. YOLO skips these cleanly in try/except.
|
||
|
||
## CUDA Version Gotcha: CUDA 13.0 + Pascal GPUs
|
||
|
||
**Symptom**: PyTorch with CUDA 13.0 (cu130) wheels installs fine, `torch.cuda.is_available()` returns True, but `model(image)` fails with:
|
||
```
|
||
RuntimeError: GET was unable to find an engine to execute this computation
|
||
```
|
||
|
||
**Root cause**: CUDA 13.0 dropped support for CC 6.x compute. Memory/mgmt functions (cudaMalloc, cudaMemcpy) work, but actual compute kernels (conv2d) fail because CUBLAS 13.0 has no binary for Pascal's architecture.
|
||
|
||
**Fix**: Use CUDA 12.6 wheels instead (cu126). Forward-compatible with driver 580:
|
||
```bash
|
||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
|
||
```
|
||
Verified on GTX 1050 Ti, driver 580.159.03.
|
||
|
||
## Flattening Sorted Photos Into a Single Folder
|
||
|
||
After classification and restore, photos live under Immich's subdirectory tree (`library/<uuid>/<year>/<month-day>/<filename>`). To flatten into a single folder:
|
||
|
||
```bash
|
||
# Create target
|
||
mkdir -p "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
|
||
|
||
# Find + flatten (handles filename collisions with _1, _2 suffix)
|
||
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type f -not -path '*/ALL Photos/*' | \
|
||
while read f; do
|
||
base=$(basename "$f")
|
||
dest="/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/$base"
|
||
if [ -f "$dest" ]; then
|
||
n=1
|
||
while [ -f "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/${base%.*}_$n.${base##*.}" ]; do n=$((n+1)); done
|
||
mv "$f" "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/${base%.*}_$n.${base##*.}"
|
||
else
|
||
mv "$f" "$dest"
|
||
fi
|
||
done
|
||
|
||
# Clean up empty dirs
|
||
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type d -empty -not -path '*/ALL Photos*' -not -path "/mnt/wd-passport/immich/NO PEOPLE PHOTOS" -delete
|
||
```
|
||
|
||
**Result from 3672 files**: all flattened in seconds. 1 filename collision (auto-renamed with `_1` suffix).
|
||
|
||
## Summary: Full Pipeline Run (16,187 Immich Photos)
|
||
|
||
```
|
||
Initial classification (1-orientation, GPU): 3,954 → NO PEOPLE
|
||
↓
|
||
EXIF-aware restore (PIL + exif_transpose): 0 restored
|
||
↓
|
||
4-orientation brute-force restore: 280 restored
|
||
↓
|
||
Final in NO PEOPLE: 3,672 flattened into ALL Photos/
|
||
```
|
||
|
||
**Total time**: ~70 min (47 min classify + 9 min EXIF restore + 9 min 4-orientation restore + ~5 min operations)
|
||
|
||
## Categorization Results (3,668 "No People" Photos)
|
||
|
||
After the full pipeline, a categorization pass was run on the 3,668 remaining photos using single-orientation YOLO to build a detection histogram:
|
||
|
||
**Result**: 2,743 photos had detections, 899 had no detections (blank/blurry/text-only screenshots), 26 errors (200MP panoramas).
|
||
|
||
**Top categories** — see the SKILL.md "Categorization by Detected Content" section for the full breakdown. The key takeaway: ~25% of sorted-out photos are blank/solid/blurry (no detections at all), and most of the detected ones are screenshots of phones/TVs/laptops or documents (books). Only ~15% are "maybe interesting" (cars, airplanes, food, clocks).
|
||
- **HEIC photos** — `pip install pi-heif` needed or they silently fail (`PIL.UnidentifiedImageError: cannot identify image file`)
|
||
- **Disk-bound** — sequential is fine, multiprocessing won't help if USB is the bottleneck
|
||
- **Kill + resume** — no resume state is saved. Script must restart from scratch. For prod, add a SQLite checkpoint file
|
||
- **GPU vs CPU** — `device="cuda:0"` requires PyTorch CUDA build. On Pascal GPUs (CC 6.x), PyTorch 2.x sometimes works with specific CUDA 12.x builds — test with `torch.cuda.is_available()` first
|
||
- **"moved: X" in logs is queued, not actual** — The classify phase tracks `len(to_move)`, displayed as "moved: 2393". Actual file moves happen in a separate phase after classification completes. During classification, the destination directory stays empty.
|
||
- **Immediate vs Batched file moves** — Two patterns. Batched (default): collect "to_move" list during classification, move in a separate phase. Log shows "moved: X" but destination is empty until the final phase. Immediate-move: `shutil.move()` inside the classification loop — destination fills in real-time, log counter is actual. Ask the user which they prefer. This configuration prefers **immediate moves**.
|
||
- **Gateway restart kills agent-launched runs** — Even if launched with nohup, the process is a child of the agent's gateway session. Restarting the gateway kills it. Use `cronjob(action="create", no_agent=True, script="...")` for durable runs.
|
||
- **Agent-launched runs waste API tokens** — The agent polls progress logs mid-turn, each poll = one LLM API call. Using `cronjob(no_agent=True)` avoids this entirely.
|
||
|
||
### Specific Error: `RuntimeError: GET was unable to find an engine to execute this computation`
|
||
|
||
**Cause**: PyTorch with CUDA 13.0 (cu130) on Pascal GPUs (CC 6.1). CUDA 13 dropped support for CC 6.x compute kernels. Memory management functions work (cudaMalloc etc.) but conv2d fails.
|
||
|
||
**Fix (modern)**: Use CUDA 12.6 wheels instead (forward-compatible with driver 580):
|
||
```bash
|
||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
|
||
```
|
||
|
||
**Fix (legacy)**: Python 3.10 + PyTorch 2.0.1+cu118 via uv (Python 3.10 + CUDA 11.8 build).
|
||
|
||
### Specific Error: `RuntimeError: Numpy is not available`
|
||
|
||
**Cause**: PyTorch 2.0.1 was compiled against numpy 1.x; `ultralytics` pulls in numpy 2.x.
|
||
|
||
**Fix**: Pin `numpy<2` after installing ultralytics.
|
||
|
||
### Specific Error: `RuntimeError: Error when binding input: There's no data transfer registered`
|
||
|
||
**Cause**: ONNX Runtime's CUDAExecutionProvider fails to load because `libcublasLt.so.12` is missing. The system has CUDA 13.0 installed, but onnxruntime-gpu was compiled against earlier CUDA.
|
||
|
||
**Fix**: Fall back to OpenVINO or use Option C (PyTorch CUDA 11.8).
|
||
|
||
### Specific Error: OOM (out of memory) with multiprocessing
|
||
|
||
**Symptom**: Workers start, process a few hundred images, then all die simultaneously. `dmesg` shows OOM killer activity.
|
||
|
||
**Cause**: `model(ap, verbose=False)` without `stream=True` accumulates all Results objects in RAM. Each worker holds results for all images it's processed so far. With 10 workers × ~200 images each = ~2000 full-resolution feature maps retained.
|
||
|
||
**Fix**: Pass `stream=True` to prevent accumulation, or reduce workers, or switch to sequential batch processing.
|
||
|
||
### Specific Warning: `WARNING ⚠️ NMS time limit 2.050s exceeded`
|
||
|
||
**Cause**: Non-maximum suppression (NMS) takes too long — typically because image decoding on a slow USB drive causes the pipeline to stall, then multiple images batch up in the worker's pipeline.
|
||
|
||
**Impact**: Self-correcting — NMS processes what it has and moves on. Not a hard error. Can indicate the USB drive is the bottleneck.
|
||
|
||
**Fix**: No action needed if results look correct. Consider reducing batch size or switching to sequential mode.
|