initial commit
This commit is contained in:
ray
@@ -0,0 +1,849 @@
|
||||
---
|
||||
name: computer-vision-inference
|
||||
description: Run YOLO and other CV models on self-hosted servers for photo/video classification. Covers GPU compatibility checks, OpenVINO fallback, batch processing patterns, and integration with media servers (Immich).
|
||||
---
|
||||
|
||||
# Computer Vision Inference on Self-Hosted Infrastructure
|
||||
|
||||
## User Preference: Plan Before Executing
|
||||
|
||||
When the user asks you to do a task, **stop and think** before running any command. Don't default to the first approach that comes to mind. Evaluate the most efficient way: consider the bottleneck (disk I/O, GPU, CPU), the right tools (dcraw, OpenVINO, CUDA), and whether the plan could fail. Getting the approach right upfront saves hours of rework. This applies to every task, not just CV — it's a general approach preference.
|
||||
|
||||
Run YOLO object detection on large photo collections (15k–50k+ images) on a homelab server.
|
||||
|
||||
## GPU Compatibility Check (First Step)
|
||||
|
||||
Before installing anything, verify the GPU can actually run modern ML frameworks:
|
||||
|
||||
```bash
|
||||
# Check GPU and compute capability (CC)
|
||||
nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader
|
||||
# OR
|
||||
nvidia-smi --query-gpu=name --format=csv,noheader
|
||||
# Check CC at: https://developer.nvidia.com/cuda-gpus
|
||||
```
|
||||
|
||||
**CC compatibility table:**
|
||||
| Architecture | CC Range | PyTorch CUDA 12.x | CUDA 13.0+ | Notes |
|
||||
|---|---|---|---|---|
|
||||
| Turing (RTX 20xx) | 7.5 | ✅ Supported | ✅ Supported | Minimum for modern PyTorch |
|
||||
| Ampere (RTX 30xx) | 8.x | ✅ Supported | ✅ Supported | |
|
||||
| Ada Lovelace (RTX 40xx) | 8.9 | ✅ Supported | ✅ Supported | |
|
||||
| **Pascal (GTX 10xx)** | **6.x** | ✅ Works (CUDA 12.4/12.6) | ❌ **Dropped** | sm_61 removed from CUDA 13.0 kernels |
|
||||
|
||||
**⚠️ Python 3.14 trap (Ubuntu 26.04):** The system Python 3.14 ships only PyTorch 2.12.0 wheels, which are compiled for CUDA 13.0. Pascal GPUs get `no kernel image is available` errors. **Fix:** install Python 3.12 from deadsnakes PPA and use CUDA 12.4 PyTorch wheels.
|
||||
|
||||
### YOLO Model Sizing by VRAM
|
||||
|
||||
| Model | Params | VRAM (batch=2, 640px) | Min GPU |
|
||||
|-------|--------|----------------------|---------|
|
||||
| YOLOv8n | 3.2M | ~400 MB | 2 GB |
|
||||
| YOLOv8s | 11.2M | ~600 MB | 2 GB |
|
||||
| YOLOv8m | 25.9M | ~1.2 GB | 3 GB |
|
||||
| **YOLOv8l** | 43.7M | ~2 GB | 4 GB |
|
||||
| **YOLOv8x** | 68.2M | ~3.5 GB | 6 GB |
|
||||
|
||||
**4GB card (GTX 1050 Ti / 1060):** Use YOLOv8s (safe) or YOLOv8m (tight but works). YOLOv8x will OOM at batch 2+. YOLOv8n is fastest but less accurate on small objects.
|
||||
|
||||
## Installation
|
||||
|
||||
### Option A: OpenVINO (for Intel CPUs or Pascal+ GPUs)
|
||||
|
||||
```bash
|
||||
# Create venv on a partition with space (not /tmp)
|
||||
sudo mkdir -p /mnt/storage/yolo_venv
|
||||
sudo chown -R $USER:$USER /mnt/storage/yolo_venv
|
||||
python3 -m venv /mnt/storage/yolo_venv
|
||||
/mnt/storage/yolo_venv/bin/pip install ultralytics
|
||||
|
||||
# Export YOLO model to OpenVINO format
|
||||
/mnt/storage/yolo_venv/bin/python3 -c "
|
||||
from ultralytics import YOLO
|
||||
model = YOLO('yolov8n.pt')
|
||||
model.export(format='openvino', device='cpu')
|
||||
"
|
||||
# Saved as ~/yolov8n_openvino_model/
|
||||
```
|
||||
|
||||
### Option B: PyTorch CUDA (for CC ≥ 7.5 GPUs)
|
||||
|
||||
```bash
|
||||
python3 -m venv /mnt/storage/yolo_venv
|
||||
/mnt/storage/yolo_venv/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||||
/mnt/storage/yolo_venv/bin/pip install ultralytics
|
||||
```
|
||||
|
||||
### Option C: PyTorch CUDA for Pascal GPUs (CC 6.x -- GTX 1050 Ti, GTX 1060, etc.)
|
||||
|
||||
Pascal GPUs (CC 6.x) have no official PyTorch CUDA support in CUDA 13.0+. On Ubuntu 26.04 the system Python 3.14 ships ONLY PyTorch 2.12.0 with CUDA 13.0 — Pascal will fail with `no kernel image is available for execution on the device`.
|
||||
|
||||
**Approach 1: Python 3.12 + CUDA 12.4 (recommended for Ubuntu 26.04)**
|
||||
|
||||
Install Python 3.12 from deadsnakes PPA, create a venv, and use CUDA 12.4 PyTorch wheels:
|
||||
|
||||
```bash
|
||||
# Install Python 3.12 (Ubuntu 26.04)
|
||||
sudo add-apt-repository -y ppa:deadsnakes/ppa
|
||||
sudo apt update && sudo apt install -y python3.12 python3.12-venv
|
||||
|
||||
# Create venv and install PyTorch CUDA 12.4
|
||||
python3.12 -m venv /mnt/storage/yolo_venv_cu124
|
||||
source /mnt/storage/yolo_venv_cu124/bin/activate
|
||||
pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 \
|
||||
--index-url https://download.pytorch.org/whl/cu124
|
||||
pip install ultralytics pillow numpy
|
||||
|
||||
# Verify
|
||||
python3 -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')"
|
||||
```
|
||||
|
||||
**Approach 2: Python 3.10 + PyTorch 2.0.1 + CUDA 11.8 (legacy stable)**
|
||||
|
||||
|
||||
```bash
|
||||
# 1. Install uv (Python 3.14+ system with PEP 668? uv bypasses it)
|
||||
pip3 install uv 2>&1 || /mnt/storage/yolo_venv/bin/pip install uv
|
||||
|
||||
# 2. Install Python 3.10
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
uv python install 3.10
|
||||
|
||||
# 3. Create venv with Python 3.10
|
||||
uv venv --python 3.10 /path/to/gpu_venv
|
||||
|
||||
# 4. Install PyTorch 2.0.1+cu118 (supports CUDA 11.8 + CC 6.1)
|
||||
uv pip install --python /path/to/gpu_venv/bin/python \
|
||||
torch==2.0.1+cu118 torchvision==0.15.2+cu118 \
|
||||
--index-url https://download.pytorch.org/whl/cu118
|
||||
|
||||
# 5. Pin numpy < 2 (PyTorch 2.0.1 was compiled against numpy 1.x)
|
||||
uv pip install --python /path/to/gpu_venv/bin/python "numpy<2"
|
||||
|
||||
# 6. Install ultralytics
|
||||
uv pip install --python /path/to/gpu_venv/bin/python ultralytics
|
||||
# If pi-heif fails via uv, install pip first then use pip:
|
||||
uv pip install --python /path/to/gpu_venv/bin/python pip
|
||||
/path/to/gpu_venv/bin/pip install pi-heif
|
||||
|
||||
# 7. Verify GPU
|
||||
/path/to/gpu_venv/bin/python3 -c "
|
||||
import torch
|
||||
print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')
|
||||
t = torch.zeros(1,3,640,640).cuda()
|
||||
conv = torch.nn.Conv2d(3, 16, 3).cuda()
|
||||
print(f'CUDA compute: OK ({conv(t).shape})')
|
||||
"
|
||||
```
|
||||
|
||||
**Performance reference (GTX 1050 Ti, USB HDD):** ~4–6 img/s at batch=4. Bottleneck is disk I/O, not GPU.
|
||||
|
||||
## YOLO Inference Patterns
|
||||
|
||||
Templates available:
|
||||
- `templates/yolo_no_people_filter.py` — GPU batch scan: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).
|
||||
|
||||
### Single image (test):
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
model = YOLO("yolov8n_openvino_model/")
|
||||
results = model("/path/to/photo.jpg", verbose=False)
|
||||
for r in results:
|
||||
if r.boxes:
|
||||
for box in r.boxes:
|
||||
cls = int(box.cls[0])
|
||||
name = r.names[cls]
|
||||
conf = float(box.conf[0])
|
||||
print(f" {name}: {conf:.2f}")
|
||||
```
|
||||
|
||||
### EXIF-aware inference (fixes sideways photos with orientation flags)
|
||||
|
||||
YOLO loads images via OpenCV (`cv2.imread`) which **ignores EXIF orientation flags**. A portrait photo taken with a phone appears rotated 90° to the model, causing missed person/object detections. Fix by pre-rotating with PIL. **However, this only catches photos that HAVE EXIF orientation flags — some photos are stored with rotated pixel data and no orientation flag at all.** See the section below for the brute-force fallback.
|
||||
|
||||
```python
|
||||
from PIL import Image, ImageOps
|
||||
import numpy as np
|
||||
|
||||
def load_image_exif_aware(path: str) -> np.ndarray:
|
||||
"""Load image with EXIF rotation applied, return BGR numpy array.
|
||||
|
||||
Note: ImageOps.exif_transpose returns None if the image has
|
||||
no EXIF orientation tag. Always handle None by reloading fresh.
|
||||
"""
|
||||
with Image.open(path) as img:
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img is None:
|
||||
# No EXIF orientation data — reload fresh
|
||||
return None # Caller should fall back to brute-force rotation
|
||||
if img.mode == "RGBA":
|
||||
img = img.convert("RGB")
|
||||
elif img.mode != "RGB":
|
||||
img = img.convert("RGB")
|
||||
return np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||||
|
||||
img_np = load_image_exif_aware("/path/to/photo.jpg")
|
||||
if img_np is not None:
|
||||
results = model(img_np, verbose=False)
|
||||
```
|
||||
|
||||
**Performance**: PIL load + EXIF transpose is fast (~11 img/s on GTX 1050 Ti) — the rotate is a metadata operation, not a pixel resample.
|
||||
|
||||
## Multi-Orientation Detection (Critical for Rotated Photos)
|
||||
|
||||
YOLO's default pipeline (OpenCV `cv2.imread`) ignores EXIF orientation flags and stores pixels as-is. A portrait phone photo appears 90° rotated to the model, causing missed detections. Fixing orientation uncovered **280+ missed people photos** in a 4,000-photo corpus.
|
||||
|
||||
### Two-Layer Strategy
|
||||
|
||||
Layer 1: EXIF-aware PIL loading (fast, catches photos with orientation flags)
|
||||
Layer 2: 4-orientation brute force (definitive, catches photos stored rotated without EXIF flags)
|
||||
|
||||
| Layer | Speed | Found on 4k photos | Time |
|
||||
|-------|-------|-------------------|------|
|
||||
| EXIF-only (PIL + `ImageOps.exif_transpose`) | ~11 img/s | **0** missed photos | ~6 min |
|
||||
| 4-orientation brute force | ~4.5 img/s | **280+** missed photos | ~25 min |
|
||||
|
||||
**Key takeaway:** EXIF-only found nothing. 4-orientation found massive numbers of missed people photos. Many phones and apps (especially older Android, TikTok screenshots, and re-saved images) store rotated pixel data **without** setting the EXIF orientation flag. EXIF handling alone is insufficient.
|
||||
|
||||
### Helper: try_all_orientations()
|
||||
|
||||
The definitive approach: rotate the image array 4 ways and check each for people (class 0).
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
def try_all_orientations(ap: str, model) -> tuple[bool, str]:
|
||||
"""Check all 4 rotations of a photo for people. Returns (found_person, orientation_name)."""
|
||||
try:
|
||||
with Image.open(ap) as img:
|
||||
if img.mode != 'RGB':
|
||||
img = img.convert('RGB')
|
||||
img_np = np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||||
except Exception:
|
||||
return (False, "")
|
||||
|
||||
orientations = {
|
||||
'0°': img_np,
|
||||
'90°': np.rot90(img_np, k=3), # 90° CW
|
||||
'180°': np.rot90(img_np, k=2), # 180°
|
||||
'270°': np.rot90(img_np, k=1), # 270° CW
|
||||
}
|
||||
|
||||
for orient_name, orient_img in orientations.items():
|
||||
try:
|
||||
results = model(orient_img, device="cuda:0", verbose=False)
|
||||
except:
|
||||
continue
|
||||
for r in results:
|
||||
if r.boxes:
|
||||
for box in r.boxes:
|
||||
if int(box.cls[0]) == 0: # person
|
||||
return (True, orient_name)
|
||||
return (False, "")
|
||||
|
||||
# Usage in restore loop:
|
||||
found, orientation = try_all_orientations(ap, model)
|
||||
if found:
|
||||
print(f"Found person at {orientation}")
|
||||
shutil.move(ap, original_path)
|
||||
```
|
||||
|
||||
**Performance**: ~4.5 img/s (4× slower than single-orientation). ~25 min for 4,000 photos. RAM stays stable (tested on GTX 1050 Ti).
|
||||
|
||||
### Full dual-pass workflow
|
||||
|
||||
```python
|
||||
# Pass 1: EXIF-aware (fast pre-filter)
|
||||
for ap in moved_photos:
|
||||
img_bgr = load_with_exif(ap) # tries EXIF transpose
|
||||
if img_bgr is None:
|
||||
continue # nothing to see RAW — pass to brute force
|
||||
results = model(img_bgr, verbose=False)
|
||||
if has_person(results):
|
||||
restore(ap)
|
||||
|
||||
# Pass 2: 4-orientation (catches everything else)
|
||||
for ap in still_in_moved_photos:
|
||||
found, orient = try_all_orientations(ap, model)
|
||||
if found:
|
||||
restore(ap)
|
||||
```
|
||||
|
||||
**Practical note**: Do pass 2 even if pass 1 found nothing — on a real corpus, pass 1 found 0 but pass 2 found 280+. The EXIF orientation flag is simply not present in many stored photos.
|
||||
|
||||
### Edge cases when loading images via PIL
|
||||
|
||||
| Problem | Cause | Fix |
|
||||
|---------|-------|-----|
|
||||
| RGBA PNG | PNG screenshots with alpha channel produce 4-channel arrays | `img = img.convert("RGB")` |
|
||||
| CMYK/grayscale | Some scanned images or legacy formats | `img = img.convert("RGB")` |
|
||||
| Decompression bomb (200MP+ panoramas) | Pillow error: "Image size exceeds limit" | `PIL.Image.MAX_IMAGE_PIXELS = None` or catch `PIL.DecompressionBombError` and skip; the latter is safer |
|
||||
| HEIC/HEIF | iPhone photos | `pip install pi-heif` |
|
||||
| Nikon RAW (NEF) | DSLR RAW files from Nikon cameras — 20-24MB each, 6000x4000px | Install `dcraw` (`sudo apt install dcraw`) for command-line conversion. Use `dcraw -c -w <file.NEF>` to pipe decoded PPM to stdout. Do NOT use rawpy/LibRaw — it fails on many camera-specific NEF variants (especially Nikon D5200+ and other consumer bodies). dcraw is the gold standard. Example: `dcraw -c -w photo.NEF | convert - photo.jpg` |
|
||||
| Truncated JPEG | Corrupt or partial downloads | Wrap in try/except, skip on error |
|
||||
| Missing EXIF orientation flag | Photo pixels are rotated but no EXIF flag set | Use 4-orientation brute force detection (see multi-orientation section) |
|
||||
|
||||
### Immediate-move pattern (files moved as classified)
|
||||
|
||||
When the user wants photos moved/purged as they're processed (not batched at the end), move inside the classification loop:
|
||||
|
||||
```python
|
||||
moved_total = 0
|
||||
for i, ap in enumerate(paths):
|
||||
results = model(ap, device="cuda:0", verbose=False)
|
||||
|
||||
should_move = False
|
||||
# ... classify results ...
|
||||
|
||||
if should_move:
|
||||
try:
|
||||
dest = ap.replace(PHOTO_DIR, NO_PEOPLE_DIR)
|
||||
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||||
shutil.move(ap, dest)
|
||||
moved_total += 1
|
||||
except Exception as e:
|
||||
log(f"MOVE FAILED: {ap} -> {e}")
|
||||
```
|
||||
|
||||
**Important**: Unlike the batched pattern where "moved: X" means "queued for move", in immediate mode `moved_total` is the actual count of files already in the destination folder.
|
||||
|
||||
### Multiprocessing batch (use `if __name__ == "__main__"` guard):
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
|
||||
def worker(args):
|
||||
model_path, paths = args
|
||||
from ultralytics import YOLO
|
||||
import warnings
|
||||
warnings.filterwarnings("ignore")
|
||||
model = YOLO(model_path, task="detect")
|
||||
results = []
|
||||
for ap in paths:
|
||||
r = model(ap, verbose=False)
|
||||
results.append((ap, decision))
|
||||
return results
|
||||
|
||||
if __name__ == "__main__":
|
||||
multiprocessing.set_start_method("forkserver", force=True)
|
||||
num_workers = min(multiprocessing.cpu_count(), 8)
|
||||
chunks = [all_paths[i::num_workers] for i in range(num_workers)]
|
||||
chunk_args = [(model_path, c) for c in chunks]
|
||||
with multiprocessing.Pool(num_workers) as pool:
|
||||
all_results = pool.map(worker, chunk_args)
|
||||
```
|
||||
|
||||
## COCO Class Classification Heuristic
|
||||
|
||||
```python
|
||||
PERSON_CLASS = 0
|
||||
NATURE_CLASSES = {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 58, 77, 80}
|
||||
MOVE_CLASSES = {56, 57, 59, 60, 61, 62, 63, 64, 65, 67, 70, 71, 72, 73, 74, 75, 76, 78, 79,
|
||||
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}
|
||||
```
|
||||
|
||||
Rules: Person→keep, Nature→keep, Indoor/urban→move, No detections→check file size (<400KB→move)
|
||||
|
||||
## Integration with Immich
|
||||
|
||||
### Query asset paths from PostgreSQL:
|
||||
|
||||
```python
|
||||
DB_QUERY = '''docker exec immich_postgres psql -U postgres -d immich -t -A -c "SELECT \\\"originalPath\\\" FROM asset WHERE type='IMAGE' AND visibility='timeline' AND \\\"originalPath\\\" LIKE '/data/library/%' ORDER BY \\\"originalPath\\\";" 2>/dev/null'''
|
||||
```
|
||||
|
||||
### Map virtual paths to filesystem:
|
||||
|
||||
```python
|
||||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||||
def map_path(db_path):
|
||||
if db_path.startswith("/data/"):
|
||||
return PHOTO_BASE + "/" + db_path[6:]
|
||||
elif db_path.startswith("/external/"):
|
||||
return "/mnt/wd-passport" + db_path[8:]
|
||||
return db_path
|
||||
```
|
||||
|
||||
### Flatten sorted photos into a single folder
|
||||
|
||||
After classification, photos end up scattered across Immich's subdirectory tree (e.g., `NO PEOPLE PHOTOS/library/4defcb72-.../2026/2026-04-07/photo.jpg`). To flatten them into a single folder with no subdirectories:
|
||||
|
||||
```python
|
||||
import os, shutil
|
||||
|
||||
NO_PEOPLE = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
|
||||
TARGET = os.path.join(NO_PEOPLE, "ALL Photos")
|
||||
os.makedirs(TARGET, exist_ok=True)
|
||||
|
||||
# Find all files excluding the target itself
|
||||
files = []
|
||||
for root, dirs, fnames in os.walk(NO_PEOPLE):
|
||||
if root.startswith(TARGET):
|
||||
continue
|
||||
for f in fnames:
|
||||
files.append(os.path.join(root, f))
|
||||
|
||||
moved = 0
|
||||
collisions = 0
|
||||
for src in files:
|
||||
fname = os.path.basename(src)
|
||||
dst = os.path.join(TARGET, fname)
|
||||
if os.path.exists(dst):
|
||||
base, ext = os.path.splitext(fname)
|
||||
n = 1
|
||||
while os.path.exists(os.path.join(TARGET, f"{base}_{n}{ext}")):
|
||||
n += 1
|
||||
dst = os.path.join(TARGET, f"{base}_{n}{ext}")
|
||||
collisions += 1
|
||||
shutil.move(src, dst)
|
||||
moved += 1
|
||||
|
||||
print(f"Moved {moved} files, {collisions} renamed for collision")
|
||||
|
||||
# Clean up empty directories
|
||||
for root, dirs, fnames in os.walk(NO_PEOPLE, topdown=False):
|
||||
if root != TARGET and not os.listdir(root) and root != NO_PEOPLE:
|
||||
os.rmdir(root)
|
||||
```
|
||||
|
||||
**Expectation**: 3,600+ files flatten in seconds. Filename collisions are rare (~1 in 3,600). Collision handling appends `_1`, `_2`, etc. before the extension.
|
||||
|
||||
## Categorization by Detected Content
|
||||
|
||||
After sorting photos out (no people, no scenery), you may want to further categorize the remaining photos by what objects they actually contain. Run YOLO on every photo and build a detection histogram, then let the user decide which categories are worthwhile and which are trash.
|
||||
|
||||
### Detection Histogram Script
|
||||
|
||||
```python
|
||||
import os, numpy as np
|
||||
from PIL import Image
|
||||
from collections import Counter
|
||||
from ultralytics import YOLO
|
||||
|
||||
ALL_PHOTOS = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
|
||||
model = YOLO("yolov8n.pt")
|
||||
|
||||
# Gather all image files
|
||||
files = sorted([os.path.join(ALL_PHOTOS, f) for f in os.listdir(ALL_PHOTOS)
|
||||
if f.lower().endswith(('.jpg', '.jpeg', '.png', '.heic', '.heif'))])
|
||||
|
||||
detection_counts = Counter() # class_id -> count of photos containing it
|
||||
no_detect = 0
|
||||
|
||||
for i, ap in enumerate(files):
|
||||
if i > 0 and i % 300 == 0:
|
||||
print(f" [{i}/{len(files)}] ... no_detect: {no_detect}", flush=True)
|
||||
|
||||
# Load with PIL (handles EXIF + RGBA)
|
||||
with Image.open(ap) as img:
|
||||
if img.mode != 'RGB':
|
||||
img = img.convert('RGB')
|
||||
img_np = np.array(img)[:, :, ::-1]
|
||||
|
||||
results = model(img_np, device="cuda:0", verbose=False)
|
||||
|
||||
classes_found = set()
|
||||
for r in results:
|
||||
if r.boxes:
|
||||
for box in r.boxes:
|
||||
classes_found.add(int(box.cls[0]))
|
||||
|
||||
if classes_found:
|
||||
for c in classes_found:
|
||||
detection_counts[c] += 1
|
||||
else:
|
||||
no_detect += 1
|
||||
|
||||
# Print top results
|
||||
CLASS_NAMES = {0:"person",1:"bicycle",2:"car",3:"motorcycle",15:"cat",16:"dog",
|
||||
39:"bottle",41:"cup",56:"chair",62:"tv",63:"laptop",67:"cell phone",
|
||||
73:"book",74:"clock"} # add more as needed
|
||||
for cls_id, count in detection_counts.most_common(20):
|
||||
name = CLASS_NAMES.get(cls_id, f"c{cls_id}")
|
||||
print(f" {name:20s}: {count:4d} photos ({count/len(files)*100:.1f}%)")
|
||||
print(f" {'no detection':20s}: {no_detect:4d} photos ({no_detect/len(files)*100:.1f}%)")
|
||||
```
|
||||
|
||||
### Real-World Detection Distribution (3,668 photos sorted out as "no people/scenery")
|
||||
|
||||
| Object | Photos | % | Likely |
|
||||
|--------|--------|---|--------|
|
||||
| book | 668 | 18% | Documents, receipts, textbooks |
|
||||
| cell phone | 342 | 9% | Phone screenshots |
|
||||
| tv/monitor | 320 | 9% | Screen captures |
|
||||
| laptop | 274 | 7% | Computer screenshots |
|
||||
| car | 265 | 7% | Vehicle photos |
|
||||
| refrigerator | 218 | 6% | Fridge contents |
|
||||
| clock | 128 | 3% | Timestamps |
|
||||
| chair | 102 | 3% | Random furniture |
|
||||
| dining table | 74 | 2% | Meals |
|
||||
| toilet | 74 | 2% | Bathroom shots |
|
||||
| airplane | 54 | 1.5% | Travel |
|
||||
| No detection | 899 | 25% | Blank/blurry/text-only |
|
||||
|
||||
### Sorting Into Subfolders by Category
|
||||
|
||||
After the histogram gives you a picture of what's there, present it to the user and ask which categories to keep vs discard. Then sort:
|
||||
|
||||
```python
|
||||
import os, shutil
|
||||
|
||||
TRASH_CLASSES = {62, 63, 67, 73} # tv, laptop, cell phone, book
|
||||
TRASH_DIR = os.path.join(ALL_PHOTOS, "..", "Trash Screenshots")
|
||||
os.makedirs(TRASH_DIR, exist_ok=True)
|
||||
|
||||
KEEP_CLASSES = {2, 5, 4, 39, 41, 74} # car, bus, airplane, bottle, cup, clock
|
||||
KEEP_DIR = os.path.join(ALL_PHOTOS, "..", "Maybe Keep")
|
||||
os.makedirs(KEEP_DIR, exist_ok=True)
|
||||
|
||||
for fname in os.listdir(ALL_PHOTOS):
|
||||
ap = os.path.join(ALL_PHOTOS, fname)
|
||||
if not os.path.isfile(ap):
|
||||
continue
|
||||
# ... check fname against a pre-computed CSV of classes ...
|
||||
shutil.move(ap, dest)
|
||||
```
|
||||
|
||||
**Practical tip**: Instead of re-running YOLO on every photo, save the classification results to a CSV during the detection histogram pass. Then use the CSV to sort — much faster:
|
||||
|
||||
```python
|
||||
# During detection pass, write:
|
||||
# all_photo_categories.csv
|
||||
# filename,detected_classes
|
||||
with open("all_photo_categories.csv", "w") as f:
|
||||
f.write("filename,detected_classes\n")
|
||||
for fname, classes in photo_classes.items():
|
||||
names = ";".join(sorted(CLASS_NAMES.get(c, f"c{c}") for c in classes))
|
||||
f.write(f"{fname},{names}\n")
|
||||
# Also record no-detection photos
|
||||
for fname in no_detect_files:
|
||||
f.write(f"{fname},none\n")
|
||||
```
|
||||
|
||||
Then sort using the CSV — instant classification lookup, no GPU needed.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **HEIC photos**: `pip install pi-heif` for HEIC/HEIF support
|
||||
- **Slow USB drives**: Use PostgreSQL DB query instead of filesystem `find`/`walk` on external drives with 20k+ files
|
||||
- **Docker bridge IP loss**: If bridge lacks gateway IP, do full `docker compose down && docker compose up -d`
|
||||
- **OpenVINO batching**: LATENCY mode won't batch — use multiprocessing for parallelism
|
||||
- **Venv placement**: PyTorch+ultralytics is ~2-3GB. Install on a data partition, not `/tmp`
|
||||
- **Multiprocessing**: Must use `if __name__ == "__main__"` guard with `set_start_method("forkserver")` in Python 3.14+
|
||||
- **`stream=True` to prevent OOM**: Without `stream=True`, YOLO holds all results in RAM across the entire run. For 15k+ images, each result with boxes metadata accumulates until the process is killed by OOM killer. Always pass `stream=True` to `model(ap, verbose=False, stream=True)` when iterating through many images. Note: `stream=True` returns a generator; materialize it with `list()` or iterate immediately.
|
||||
- **Zombie GPU processes**: Killing a YOLO script (Ctrl+C, `kill`, or OOM crash) often leaves a zombie Python process holding all VRAM. The GPU shows 4024MiB / 4096MiB used but no visible process. Run `nvidia-smi` to find the PID, then `kill -9 <pid>`. Always check `nvidia-smi` after a failed run — the GPU won't be usable until the zombie is cleared.
|
||||
- **Cascading CUDA failures after zombie**: If a new YOLO run immediately OOMs on a model that previously fit (e.g. YOLOv8s on 4GB), check `nvidia-smi` for a ghost process from a prior run. The "out of memory" message is correct — the GPU IS full, just by a process that appears dead.
|
||||
- **Corrupt file batch fallback trap**: When a YOLO batch fails due to corrupt images, the naive fallback re-tries each image individually on GPU at reduced resolution. On a 4GB card with dozens of corrupt files (common in recovery data), this can stall the pipeline for hours — each retry re-loads the model, allocates CUDA context, and fails. **Fix**: treat batch-level failures as a corrupt batch. Move all images in the failed batch directly to the destination without any GPU retry. If the user asks to classify corrupt images separately, use PIL to check for readability first (fast CPU operation), then only run YOLO on readable files.
|
||||
- **Background output buffering via SSH**: When launching a long-running Python script via `terminal(background=true)` over SSH, standard output is often completely invisible — the SSH pipe buffers aggressively and the process viewer shows empty output for minutes. **Fix**: redirect stdout+stderr to a log file on the remote server (`> /tmp/script.log 2>&1`), then monitor the log file with `read_file()`. Also use `python3 -u` (unbuffered mode) and `flush=True` on all print() calls.
|
||||
- **NumPy version mismatch**: PyTorch 2.0.1 was compiled against numpy 1.x. Installing ultralytics pulls in numpy 2.x, causing `RuntimeError: Numpy is not available` at inference time. Pin `numpy<2` in the venv after installing everything else.
|
||||
- **`setsid` for background isolation**: `nohup` is insufficient for SSH-launched background processes — `setsid` detaches from the terminal session entirely, preventing SIGHUP from the SSH closure. Use: `setsid python /tmp/script.py > /home/ray/log 2>&1 &`
|
||||
|
||||
## GPU Memory FAQ
|
||||
|
||||
Users frequently ask "why is my GPU only using X MB when it has Y GB available?" during YOLO inference:
|
||||
|
||||
### Why VRAM usage stays low (e.g. 432MB / 4GB)
|
||||
|
||||
| Factor | Why |
|
||||
|--------|-----|
|
||||
| **Model size** | YOLOv8n (nano) is ~6.3MB. Weights + CUDA overhead = ~200–500MB VRAM. YOLOv8x would use ~1–2GB |
|
||||
| **Batch size = 1** | Each frame is processed and discarded before the next. No gradient history, no activation cache (those are training-only) |
|
||||
| **Inference ≠ Training** | Training batches many images, backpropagates through the full graph, and caches activations for gradient computation — that's what fills VRAM. Inference is forward-pass-only |
|
||||
| **CUDA context floor** | PyTorch reserves ~300–500MB of VRAM just for the CUDA runtime context regardless of model size |
|
||||
|
||||
The 1050 Ti's 4GB is sitting idle because the workload doesn't need more — not because something's misconfigured. YOLOv8n at batch-1 on a single image simply doesn't fill a 4GB buffer.
|
||||
|
||||
### Why system RAM is high even with GPU inference
|
||||
|
||||
If a YOLO process shows 3.7GB RSS in `ps aux` but only 432MB in `nvidia-smi`:
|
||||
|
||||
| Layer | RAM |
|
||||
|-------|-----|
|
||||
| Python + PyTorch + Ultralytics + CUDA libs loaded in memory | ~1.5 GB |
|
||||
| Image decode buffers (reads JPEGs from disk into pixel arrays) | varies |
|
||||
| PyTorch pre-allocated CPU tensor arena | ~500 MB |
|
||||
| CUDA driver / runtime bookkeeping on CPU side | ~200 MB |
|
||||
| Python overhead (file path list, results dicts for 15k+ images) | ~200 MB |
|
||||
|
||||
The GPU is the calculator, but the CPU is the assistant that reads files from disk, decodes JPEGs, normalizes pixels, and copies tensors over. The assistant needs a desk (RAM) to work at.
|
||||
|
||||
### Sequential Batch Processing (for disk-bound pipelines)
|
||||
|
||||
## Comparing NEF (RAW) vs JPG Pairs with YOLO
|
||||
|
||||
Useful when you need to verify that paired RAW+JPEG files (same base filename in a Nikon camera's dual-save setup) are actually the same photo. The JPEG is the camera's processed output, the NEF is the sensor data — they should be the same image with different processing.
|
||||
|
||||
### Pitfall: rawpy/LibRaw fails on many consumer NEFs
|
||||
|
||||
`rawpy` (Python binding for LibRaw) produces "data corrupted" errors on many Nikon D5200 and similar consumer-body NEF files. These errors are **false positives** — the files open fine in Lightroom, Nikon's software, and `dcraw`. Always fall back to dcraw for NEF conversion.
|
||||
|
||||
### dcraw + YOLO comparison workflow
|
||||
|
||||
```python
|
||||
import subprocess, io
|
||||
from PIL import Image
|
||||
|
||||
# Convert NEF to PPM via dcraw (piped to stdout)
|
||||
r = subprocess.run(['dcraw', '-c', '-w', str(nef_path)],
|
||||
capture_output=True, timeout=60)
|
||||
if r.returncode != 0:
|
||||
raise Exception(f"dcraw failed: {r.stderr.decode()[:80]}")
|
||||
|
||||
# Read PPM from stdout
|
||||
nef_img = Image.open(io.BytesIO(r.stdout)).convert('RGB')
|
||||
jpg_img = Image.open(jpg_path).convert('RGB')
|
||||
|
||||
# Resize both to same dimensions for comparison
|
||||
target = (640, 480)
|
||||
nef_r = nef_img.resize(target, Image.LANCZOS)
|
||||
jpg_r = jpg_img.resize(target, Image.LANCZOS)
|
||||
|
||||
# Run YOLO on CPU
|
||||
from ultralytics import YOLO
|
||||
model = YOLO('yolov8n.pt')
|
||||
jr = model(jpg_r, device='cpu', verbose=False)
|
||||
nr = model(nef_r, device='cpu', verbose=False)
|
||||
|
||||
# Compare detected classes
|
||||
jc = sorted([model.names[int(b.cls)] for b in jr[0].boxes])
|
||||
nc = sorted([model.names[int(b.cls)] for b in nr[0].boxes])
|
||||
|
||||
if jc == nc:
|
||||
print("Same photo — identical YOLO detections")
|
||||
else:
|
||||
print(f"Different detections: JPG={jc} vs NEF={nc}")
|
||||
```
|
||||
|
||||
### Expected results
|
||||
|
||||
The NEF version typically detects **more objects** than the JPEG version because it has the full sensor dynamic range (24MP, deeper shadows/highlights). A JPG that sees 4 people might detect 8 in the NEF, all in shadow areas the JPG crushed. This **confirms they're the same photo** with richer data in the RAW version — not different photos.
|
||||
|
||||
### NEF size reference
|
||||
|
||||
| Camera | NEF size | JPEG size | Resolution |
|
||||
|--------|----------|-----------|------------|
|
||||
| Nikon D5200 | ~23 MB | ~2-8 MB | 6036 × 4020 |
|
||||
|
||||
### When to use CPU vs GPU for verification
|
||||
|
||||
If the installed PyTorch version doesn't support the GPU's compute capability (common with GTX 1050 Ti / CC 6.1 on modern PyTorch CUDA 12+), force CPU:
|
||||
```python
|
||||
model(image, device='cpu', verbose=False)
|
||||
```
|
||||
CPU inference on 640×480 images with YOLOv8n is ~1-2 img/s per core. For 100 images, expect ~2-3 minutes — acceptable for verification tasks.
|
||||
|
||||
When the bottleneck is disk I/O (reading from a USB HDD), not GPU compute, multiprocessing doesn't help much. A simple sequential loop is simpler and more predictable:
|
||||
|
||||
```python
|
||||
from ultralytics import YOLO
|
||||
model = YOLO("yolov8n.pt")
|
||||
|
||||
counts = defaultdict(int)
|
||||
to_move = []
|
||||
t_start = time.time()
|
||||
|
||||
for i, ap in enumerate(all_paths):
|
||||
if i > 0 and i % 500 == 0:
|
||||
elapsed = time.time() - t_start
|
||||
rate = i / elapsed
|
||||
eta = (len(all_paths) - i) / rate / 60
|
||||
print(f" [{i}/{len(all_paths)}] {rate:.1f} img/s, ETA {eta:.0f}min", flush=True)
|
||||
|
||||
try:
|
||||
results = model(ap, device="cuda:0", verbose=False)
|
||||
except:
|
||||
continue
|
||||
|
||||
# classify results...
|
||||
```
|
||||
|
||||
**Performance reference (GTX 1050 Ti, USB HDD):** ~6–7 img/s on 16k images, ~40 min total for 16k photos.
|
||||
|
||||
## Running Long Inference Without Blocking the Hermes Gateway
|
||||
|
||||
**Critical:** If the agent launches a long classification script inside its conversation turn, the Telegram/Discord bot appears "typing..." and **cannot process new messages** until the turn finishes. Each agent iteration also makes an LLM API call — polling progress wastes tokens while the user is charged per-call.
|
||||
|
||||
### ❌ Anti-pattern: Agent launches script and waits
|
||||
|
||||
```
|
||||
User: "classify my photos"
|
||||
Agent: runs terminal("python classify.py") ← blocks for 40 min
|
||||
User: "what's the status?" ← queued, not seen
|
||||
...bot shows typing for 40 min...
|
||||
← agent finally responds (dozens of API calls eaten)
|
||||
```
|
||||
|
||||
### ✅ Pattern A: Durable cron job (recommended)
|
||||
|
||||
Create a cron job with `no_agent=True` — the script runs independently, stdout goes directly to Telegram. Survives gateway restart, zero token cost during execution:
|
||||
|
||||
```
|
||||
cronjob(
|
||||
action="create",
|
||||
schedule="now",
|
||||
script="/home/ray/yolo_classify.py",
|
||||
no_agent=True,
|
||||
deliver="origin"
|
||||
)
|
||||
```
|
||||
|
||||
### ✅ Pattern B: terminal(background=true) + log file + end turn
|
||||
|
||||
Launch with `background=true`, redirect stdout to a log file (SSH pipe buffers aggressively — without this, output is invisible), and immediately end the turn. Bot becomes responsive but **does not survive gateway restart** — child processes are killed when the gateway restarts:
|
||||
|
||||
```bash
|
||||
# Redirect to a log file to avoid SSH pipe buffering
|
||||
terminal(command="bash -c 'source venv/bin/activate && python3 -u classify.py > /tmp/classify.log 2>&1'", background=true, notify_on_complete=true)
|
||||
# Monitor progress via read_file("/tmp/classify.log")
|
||||
```
|
||||
|
||||
**Note:** Always use `python3 -u` (unbuffered) and `flush=True` on print() calls, even with log redirection. Without `-u`, Python's own output buffering delays log writes and `read_file()` shows stale data.
|
||||
|
||||
**This user's preference**: Always run long tasks in the background. They want to stay able to message me while work is processing — no waiting for 10-40 min tasks to finish before I respond.
|
||||
```
|
||||
|
||||
### Pitfall: "moved: X" is queued, not actually moved
|
||||
|
||||
In a 2-phase script (classify all → then move all), the log shows `moved: 2393` during classification. This is `len(to_move)` — photos **queued for moving**, not actually moved. The destination directory stays empty until the move phase runs. Label the counter clearly to avoid confusion:
|
||||
|
||||
```python
|
||||
# During classification:
|
||||
print(f" queued for move: {len(to_move)}", flush=True)
|
||||
|
||||
# During move phase:
|
||||
moved = 0
|
||||
for ap in to_move:
|
||||
shutil.move(ap, dest)
|
||||
moved += 1
|
||||
print(f" Actually moved: {moved}", flush=True)
|
||||
```
|
||||
|
||||
### Pitfall: Gateway restart kills agent-child processes
|
||||
|
||||
Agent-launched processes (even nohup'd) are children of the gateway process tree. Restarting the gateway orphans/kills them. For work that must survive restarts, always use a cron job with `no_agent=True` (Pattern A).
|
||||
|
||||
## Video Content Detection and Renaming
|
||||
|
||||
When dealing with recovered or generically-named video files (e.g., FILE000.MOV, FILE002.MOV from data recovery tools), you can auto-detect their content using YOLO on extracted frames and rename them descriptively.
|
||||
|
||||
### Workflow
|
||||
|
||||
1. Extract a single frame from each video at ~3 seconds
|
||||
2. Run YOLO on CPU (since video processing doesn't benefit from GPU for single-frame extraction)
|
||||
3. Build a descriptive filename from detected objects
|
||||
4. Handle collisions, blank frames, and edge cases
|
||||
|
||||
### Full Renaming Script
|
||||
|
||||
```python
|
||||
import os, subprocess, json
|
||||
from PIL import Image
|
||||
from ultralytics import YOLO
|
||||
|
||||
DIR = "/mnt/drive/Videos/Recovered/MOV Multimedia file"
|
||||
model = YOLO("yolov8n.pt")
|
||||
|
||||
files = sorted([f for f in os.listdir(DIR)
|
||||
if f.upper().endswith(('.MOV', '.MP4', '.3GP', '.AVI'))
|
||||
and f.upper().startswith("FILE")])
|
||||
|
||||
for i, fname in enumerate(files):
|
||||
src = os.path.join(DIR, fname)
|
||||
base, ext = os.path.splitext(fname)
|
||||
|
||||
# Extract frame at 3 seconds (fall back to 1s for short clips)
|
||||
frame_path = "/tmp/_vf.jpg"
|
||||
for ss in ["00:00:03", "00:00:01"]:
|
||||
r = subprocess.run(["ffmpeg", "-y", "-ss", ss, "-i", src,
|
||||
"-vframes", "1", "-q:v", "2", frame_path],
|
||||
capture_output=True, timeout=20)
|
||||
if os.path.exists(frame_path) and os.path.getsize(frame_path) > 1000:
|
||||
break
|
||||
|
||||
# Run YOLO on extracted frame
|
||||
classes = []
|
||||
if os.path.exists(frame_path):
|
||||
try:
|
||||
img = Image.open(frame_path)
|
||||
results = model(img, device="cpu", verbose=False)
|
||||
classes = [model.names[int(b.cls)] for b in results[0].boxes]
|
||||
os.remove(frame_path)
|
||||
except:
|
||||
pass
|
||||
|
||||
# Build descriptive name
|
||||
if classes:
|
||||
people = classes.count("person")
|
||||
other = [c for c in sorted(set(classes)) if c != "person"]
|
||||
parts = []
|
||||
if people == 1:
|
||||
parts.append("person")
|
||||
elif people > 1:
|
||||
parts.append(f"{people}_people")
|
||||
parts.extend(other[:3]) # max 3 other objects
|
||||
new_name = "_".join(parts).replace(" ", "_") + ext.lower()
|
||||
else:
|
||||
new_name = f"scene{ext.lower()}" # blank/dark/no detections
|
||||
|
||||
# Avoid filename collisions
|
||||
dst = os.path.join(DIR, new_name)
|
||||
counter = 1
|
||||
while os.path.exists(dst):
|
||||
name_no_ext = os.path.splitext(new_name)[0]
|
||||
dst = os.path.join(DIR, f"{name_no_ext}_{counter}{ext.lower()}")
|
||||
counter += 1
|
||||
|
||||
os.rename(src, dst)
|
||||
if (i+1) % 10 == 0:
|
||||
print(f" [{i+1}/{len(files)}]", flush=True)
|
||||
```
|
||||
|
||||
### Real-World Renaming Examples (80 recovered files)
|
||||
|
||||
| Old Name | New Name | Detected |
|
||||
|----------|----------|----------|
|
||||
| FILE000.MOV | `chair_clock_tv.mov` | Indoor scene with TV |
|
||||
| FILE054.MOV | `bench_car_elephant.mov` | Outdoor with animals |
|
||||
| FILE125.MOV | `5_people_cake_cell_phone_dining_table.mov` | Party/birthday |
|
||||
| FILE214.MOV | `person_dog.mov` | Person walking dog |
|
||||
| FILE021.MOV | `2_people_chair_surfboard.mov` | Beach/outdoor |
|
||||
| FILE176.MOV | `kite.mov` | Kite flying |
|
||||
| FILE095.MOV | `scene.mov` | Blank/dark frame |
|
||||
|
||||
### Expected Results Distribution
|
||||
|
||||
On 80 recovered video files from a data recovery tool:
|
||||
- **~60 files** get descriptive names (people, animals, objects detected)
|
||||
- **~20 files** become `scene.mov` (blank/dark frames — common in recovered data where keyframes didn't survive)
|
||||
- **~0-2 files** hit errors (truncated files, unreadable headers)
|
||||
|
||||
### Performance
|
||||
|
||||
- **~3-5 seconds per file** on CPU (i7-10700K): ffmpeg frame extraction (~1s) + YOLOv8n inference (~2-4s)
|
||||
- **~4-7 minutes total** for 80 files
|
||||
- GPU doesn't help here — ffmpeg decoding and single-frame inference are CPU-bound at this scale
|
||||
|
||||
### Pitfalls
|
||||
|
||||
- **Blank/dark frames at 3s**: Some recovered videos have corruption at specific timestamps. The fallback to 1s catches most of these. If both fail, the file becomes `scene.mov` — still identifiable as having no content.
|
||||
- **`.3GP` extension**: Data recovery tools often produce .3GP files alongside .MOV. These are valid video containers. Include them in the file filter.
|
||||
- **Short clips**: Videos under 3 seconds cause ffmpeg to warn about invalid timestamp but still produce a frame from the beginning. The `ss` flag gracefully clips to available duration.
|
||||
- **Collision avoidance**: Multiple files with the same detected objects (e.g., 3 different `person.mov` files) get `_1`, `_2` suffixes. The inner `while` loop checks existence and increments.
|
||||
- **Don't rename twice**: Files already renamed (no "FILE" prefix) will be skipped by the `startswith("FILE")` guard. Always keep this guard for idempotency.
|
||||
- **Broken file headers**: Truncated MOV files may cause ffmpeg to hang. Always set `timeout=20` on subprocess calls.
|
||||
|
||||
## Performance Estimates (i7-10700K, 16 threads)
|
||||
|
||||
| Engine | img/s (1 worker) | img/s (8 workers) | 25k images |
|
||||
|---|---|---|---|
|
||||
| PyTorch CPU | 0.3 | — | ~23h |
|
||||
| OpenVINO | 1.5 | ~8-10 | ~4.5h |
|
||||
| PyTorch CUDA (if supported) | 30+ | — | ~15min |
|
||||
|
||||
## Reference Files
|
||||
|
||||
- `references/immich-yolo-classifier.md` — Immich-specific YOLO classifier query + path mapping.
|
||||
- `references/immich-classification-pipeline.md` — Full Immich photo classification pipeline: DB query, EXIF-aware multi-orientation inference, immediate-move pattern, content categorization, flat-folder sorting. (Absorbed from `immich-photo-classification`.)
|
||||
- `references/video-content-renaming.md` — YOLO-based video frame detection and auto-renaming for recovered files.
|
||||
- `templates/yolo_no_people_filter.py` — GPU batch scan template: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
name: immich-photo-classification
|
||||
description: Classify, sort, and organize Immich photos using YOLO on GPU — people detection, content categorization, and trash/keep sorting.
|
||||
trigger: |
|
||||
Load when the user asks to classify Immich photos, run YOLO on their media library,
|
||||
sort photos by content, move photos out of Immich, restore photos with missed detections,
|
||||
or organize NO PEOPLE / flagged photo folders.
|
||||
Also load when the user wants to categorize or filter their photo library by content type.
|
||||
---
|
||||
|
||||
# Immich Photo Classification Pipeline
|
||||
|
||||
Full-stack workflow for classifying Immich photos with YOLOv8n on GPU, including:
|
||||
- People/nature detection to filter non-people photos
|
||||
- EXIF-aware re-checking with multi-orientation inference
|
||||
- Content-based categorization (documents, screenshots, food, vehicles, etc.)
|
||||
- Sorting into Trash/Keep folders
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **yolov8s.pt or yolov8n.pt** — downloaded at first run (ultralytics auto-downloads)
|
||||
- **Python venv with** `ultralytics` + `pillow` + `numpy` + `torch` (CUDA)
|
||||
- **Immich server** running with PostgreSQL accessible via `docker exec`
|
||||
- **For GTX 1050 Ti (Pascal, CC 6.1) on Ubuntu 26.04:** MUST use Python 3.12 + CUDA 12.4. The system Python 3.14 ships PyTorch 2.12 with CUDA 13.0 which dropped Pascal support.
|
||||
- Install Python 3.12: `sudo add-apt-repository -y ppa:deadsnakes/ppa && sudo apt update && sudo apt install -y python3.12 python3.12-venv`
|
||||
- Create venv: `python3.12 -m venv /mnt/storage/yolo_venv_cu124`
|
||||
- Install: `pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 --index-url https://download.pytorch.org/whl/cu124 && pip install ultralytics`
|
||||
- On 4GB VRAM: use YOLOv8s (11M params). YOLOv8x (68M) will OOM.
|
||||
- **After any crash or killed run:** check `nvidia-smi` for zombie GPU processes and `kill -9` them before restarting.
|
||||
|
||||
## Step-by-Step
|
||||
|
||||
### 1. Query Immich Database
|
||||
|
||||
```python
|
||||
cmd = '''docker exec immich_postgres psql -U postgres -d immich -t -A -c "SELECT \\\"originalPath\\\" FROM asset WHERE type='IMAGE' AND visibility='timeline' AND \\\"originalPath\\\" LIKE '/data/library/%' ORDER BY \\\"originalPath\\\";" 2>/dev/null'''
|
||||
```
|
||||
|
||||
Map the DB paths to filesystem:
|
||||
```python
|
||||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||||
ap = PHOTO_BASE + "/" + p[6:] # strip '/data/' prefix
|
||||
```
|
||||
|
||||
### 2. GPU Inference — CRMICAL: Fix Orientation First
|
||||
|
||||
**DO NOT pass file paths directly to YOLO for phone photos.**
|
||||
YOLO uses OpenCV (`cv2.imread()`) which does NOT read EXIF orientation flags.
|
||||
Portrait/rotated phone photos will be analyzed sideways and people will be missed.
|
||||
|
||||
**CORRECT approach — try ALL 4 orientations:**
|
||||
|
||||
```python
|
||||
from PIL import Image, ImageOps
|
||||
import numpy as np
|
||||
|
||||
with Image.open(ap) as img:
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img is None:
|
||||
img = Image.open(ap) # reload without EXIF
|
||||
if img.mode != 'RGB':
|
||||
img = img.convert('RGB') # handles RGBA, CMYK, etc.
|
||||
img_bgr = np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||||
|
||||
# Try all 4 orientations
|
||||
orientations = {
|
||||
'0°': img_bgr,
|
||||
'90°': np.rot90(img_bgr, k=3),
|
||||
'180°': np.rot90(img_bgr, k=2),
|
||||
'270°': np.rot90(img_bgr, k=1),
|
||||
}
|
||||
```
|
||||
|
||||
### 3. Move Files Immediately
|
||||
|
||||
**DO NOT batch moves at the end.** The user wants files moved as soon as they're flagged:
|
||||
|
||||
```python
|
||||
shutil.move(ap, dest) # move immediately when flagged
|
||||
```
|
||||
|
||||
### 4. Content Categorization
|
||||
|
||||
After classifying all photos, categorize by detected objects using COCO classes.
|
||||
|
||||
**Trash classes** (likely screenshots/boring):
|
||||
- `cell phone` (67) — phone screenshots
|
||||
- `tv` (62) — TV/monitor captures
|
||||
- `laptop` (63) — computer screenshots
|
||||
- `keyboard` (66), `mouse` (64), `remote` (65)
|
||||
- `toilet` (61), `chair` (56), `couch` (57), `bed` (59)
|
||||
- `refrigerator` (72), `microwave` (68), `oven` (69), `sink` (71)
|
||||
- `hair drier` (78), `toothbrush` (79), `scissors` (76)
|
||||
- No detections / blank images
|
||||
|
||||
**Keep classes** (potentially important):
|
||||
- `book` (73) — documents, receipts
|
||||
- `car` (2), `truck` (7), `airplane` (4) — vehicles/travel
|
||||
- `dining table` (60), `cup` (41), `bottle` (39) — food/meals
|
||||
- `clock` (74) — time-stamped events
|
||||
- Animals: `dog` (16), `cat` (15), `bird` (14)
|
||||
- `potted plant` (58), `vase` (75)
|
||||
- Sports: `sports ball` (32), `baseball bat` (34)
|
||||
- Everything else not in Trash set
|
||||
|
||||
Logic: if ANY detected class is a Keep class → Keep folder.
|
||||
If ALL detected classes are Trash classes OR no detections → Trash folder.
|
||||
|
||||
### 5. Sort into Flat Folders
|
||||
|
||||
Use a flat structure (no subdirectories) for the sorted output:
|
||||
|
||||
```python
|
||||
ALL_PHOTOS → Trash/ and Keep/
|
||||
```
|
||||
|
||||
Handle filename collisions with numbered suffixes (`_1`, `_2`, etc.).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **RGBA images** (PNGs with alpha): convert to RGB before passing to YOLO, otherwise: `expected input[1, 4, 640, 416] to have 3 channels`
|
||||
- **Decompression bomb protection**: Pillow rejects images >178M pixels by default. These are typically 200MP phone panorama stitches. Log and skip them.
|
||||
- **RAM accumulation**: `model(source)` without `stream=True` accumulates results. For single-image inference in a loop this is manageable (results go out of scope), but for large batches consider `stream=True`.
|
||||
- **GPU compatibility**: GTX 1050 Ti (Pascal, CC 6.1) needs PyTorch with CUDA 12.x. CUDA 13.0+ PyTorch may fail. Use `/home/ray/yolo_venv_cu124/bin/python` (Python 3.12 + PyTorch 2.5.1+cu124) on rayserver.
|
||||
- **`ImageOps.exif_transpose()` can return None**: Always handle this case — reload the image without EXIF fallback.
|
||||
- **File naming collisions**: When flattening subdirectories, use numbered suffixes to avoid overwriting.
|
||||
- **Empty directories**: After moving all files out of subdirectories, clean them up with `find ... -type d -empty -delete`.
|
||||
@@ -0,0 +1,243 @@
|
||||
# Immich YOLO Photo Classifier — Production Script
|
||||
|
||||
A working sequential YOLO classification pipeline for Immich photos on a self-hosted server with GTX 1050 Ti.
|
||||
|
||||
## Script
|
||||
|
||||
Saved at `/tmp/yolo_gpu_run.py` on rayserver. Key design:
|
||||
|
||||
### Workflow
|
||||
1. **Query Immich DB** via `docker exec immich_postgres psql` — pulls all timeline IMAGE assets
|
||||
2. **Map virtual paths** — `/data/library/...` → `/mnt/wd-passport/immich/photos/...`
|
||||
3. **Classify with YOLOv8n** on CUDA — processes one image at a time, logs every 500
|
||||
4. **Move non-people, non-scenic photos** to `/mnt/wd-passport/immich/NO PEOPLE PHOTOS`
|
||||
|
||||
### Classification heuristic (COCO classes)
|
||||
```
|
||||
PERSON_CLASS = 0 → keep
|
||||
NATURE_CLASSES = {16,17,18,19,20,21,22,23,24, → keep
|
||||
25,26,27,58,77,80}
|
||||
MOVE_CLASSES = {56,57,59–65,67,70–80,1–15} → move (indoor/urban/vehicle/animals)
|
||||
No detections + file < 400KB → move
|
||||
No detections + file ≥ 400KB → keep as scenic
|
||||
```
|
||||
|
||||
### Running it
|
||||
Script is launched with `nohup` so it survives terminal closure:
|
||||
```bash
|
||||
nohup ~/yolo_venv_cu126/bin/python3 /tmp/yolo_gpu_run.py > ~/yolo_run.log 2>&1
|
||||
```
|
||||
|
||||
Check progress: `tail -5 ~/yolo_run.log`
|
||||
|
||||
### Performance (GTX 1050 Ti, USB HDD)
|
||||
- **16,187 images** in ~40 min (6–7 img/s)
|
||||
- **1,875 photos** moved in a test run (12%)
|
||||
- Bottleneck is disk I/O (WD Passport USB HDD), not GPU
|
||||
|
||||
### EXIF-Aware Restore (fixes sideways missed detections — NOT always sufficient)
|
||||
|
||||
When YOLO misses people in rotated/portrait photos, first try PIL-based EXIF rotation. **But note: EXIF-only is NOT always sufficient.** Some photos are stored with rotated pixel data and no EXIF orientation flag at all. The brute-force 4-orientation approach below covers all cases.
|
||||
|
||||
#### Approach A: EXIF rotation (fast, catches flagged rotations)
|
||||
|
||||
```python
|
||||
from PIL import Image, ImageOps
|
||||
import numpy as np
|
||||
from ultralytics import YOLO
|
||||
|
||||
model = YOLO("yolov8n.pt")
|
||||
NO_PEOPLE_DIR = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
|
||||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||||
|
||||
for ap in files:
|
||||
# Load with EXIF rotation applied
|
||||
with Image.open(ap) as img:
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img is None:
|
||||
continue
|
||||
if img.mode != "RGB":
|
||||
img = img.convert("RGB")
|
||||
img_np = np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||||
|
||||
results = model(img_np, device="cuda:0", verbose=False)
|
||||
|
||||
has_person = any(
|
||||
int(box.cls[0]) == 0
|
||||
for r in results if r.boxes
|
||||
for box in r.boxes
|
||||
)
|
||||
|
||||
if has_person:
|
||||
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
|
||||
os.makedirs(os.path.dirname(orig), exist_ok=True)
|
||||
shutil.move(ap, orig)
|
||||
```
|
||||
|
||||
**Result from 3951 photos**: 0 restores — EXIF rotation found no additional people.
|
||||
|
||||
#### ✅ Approach B: 4-Orientation brute-force (definitive, catches all rotation cases)
|
||||
|
||||
When EXIF-only finds nothing but the user insists people exist in sideways photos, switch to trying all 4 orientations. This is the **definitive** approach:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
def try_all_orientations(ap: str, model) -> tuple[bool, str]:
|
||||
with Image.open(ap) as img:
|
||||
if img.mode != 'RGB':
|
||||
img = img.convert('RGB')
|
||||
img_np = np.array(img)[:, :, ::-1]
|
||||
|
||||
orientations = {
|
||||
'0°': img_np,
|
||||
'90°': np.rot90(img_np, k=3),
|
||||
'180°': np.rot90(img_np, k=2),
|
||||
'270°': np.rot90(img_np, k=1),
|
||||
}
|
||||
|
||||
for orient_name, orient_img in orientations.items():
|
||||
results = model(orient_img, device="cuda:0", verbose=False)
|
||||
for r in results:
|
||||
if r.boxes:
|
||||
for box in r.boxes:
|
||||
if int(box.cls[0]) == 0:
|
||||
return (True, orient_name)
|
||||
return (False, "")
|
||||
|
||||
# Restore loop
|
||||
restored = 0
|
||||
for ap in files:
|
||||
found, orientation = try_all_orientations(ap, model)
|
||||
if found:
|
||||
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
|
||||
os.makedirs(os.path.dirname(orig), exist_ok=True)
|
||||
shutil.move(ap, orig)
|
||||
restored += 1
|
||||
print(f"RESTORED ({orientation}): {os.path.basename(ap)}")
|
||||
```
|
||||
|
||||
**Result from 3951 photos**: **280 restored** (primarily at 90° and 270°). Most missing people were in photos taken on phones that store portrait pixel data without EXIF orientation flags.
|
||||
|
||||
**Performance**: ~4.5 img/s, ~25 min for 4k photos, RAM stable at 14%.
|
||||
|
||||
## Image Loading Edge Cases (PIL path)
|
||||
|
||||
When passing a numpy array (from PIL) instead of a file path to YOLO, these issues arise:
|
||||
|
||||
- **`ImageOps.exif_transpose` returns None** — This happens when the image has no EXIF orientation tag at all (common with screenshots, downloaded images, re-saved files). The old pattern `if img is None: continue` **silently skips these images** without counting them as errors or "still empty". Fix: reload the image fresh without EXIF if None, or fall through to 4-orientation brute force.
|
||||
- **RGBA PNG** (4 channels): `RuntimeError: expected input[1, 4, 640, 416] to have 3 channels, but got 4 channels instead`. Fix: `img = img.convert("RGB")`.
|
||||
- **Decompression bomb**: `Image size (199756800 pixels) exceeds limit of 178956970 pixels`. Large panoramas/stitches (~200MP) trigger Pillow's safety limit. Fix: `PIL.Image.MAX_IMAGE_PIXELS = None` before loading, or skip large files.
|
||||
- **Corrupt JPEG**: `Invalid SOS parameters for sequential JPEG` / `Corrupt JPEG data: 1 extraneous bytes before marker`. YOLO skips these cleanly in try/except.
|
||||
|
||||
## CUDA Version Gotcha: CUDA 13.0 + Pascal GPUs
|
||||
|
||||
**Symptom**: PyTorch with CUDA 13.0 (cu130) wheels installs fine, `torch.cuda.is_available()` returns True, but `model(image)` fails with:
|
||||
```
|
||||
RuntimeError: GET was unable to find an engine to execute this computation
|
||||
```
|
||||
|
||||
**Root cause**: CUDA 13.0 dropped support for CC 6.x compute. Memory/mgmt functions (cudaMalloc, cudaMemcpy) work, but actual compute kernels (conv2d) fail because CUBLAS 13.0 has no binary for Pascal's architecture.
|
||||
|
||||
**Fix**: Use CUDA 12.6 wheels instead (cu126). Forward-compatible with driver 580:
|
||||
```bash
|
||||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
|
||||
```
|
||||
Verified on GTX 1050 Ti, driver 580.159.03.
|
||||
|
||||
## Flattening Sorted Photos Into a Single Folder
|
||||
|
||||
After classification and restore, photos live under Immich's subdirectory tree (`library/<uuid>/<year>/<month-day>/<filename>`). To flatten into a single folder:
|
||||
|
||||
```bash
|
||||
# Create target
|
||||
mkdir -p "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
|
||||
|
||||
# Find + flatten (handles filename collisions with _1, _2 suffix)
|
||||
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type f -not -path '*/ALL Photos/*' | \
|
||||
while read f; do
|
||||
base=$(basename "$f")
|
||||
dest="/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/$base"
|
||||
if [ -f "$dest" ]; then
|
||||
n=1
|
||||
while [ -f "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/${base%.*}_$n.${base##*.}" ]; do n=$((n+1)); done
|
||||
mv "$f" "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/${base%.*}_$n.${base##*.}"
|
||||
else
|
||||
mv "$f" "$dest"
|
||||
fi
|
||||
done
|
||||
|
||||
# Clean up empty dirs
|
||||
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type d -empty -not -path '*/ALL Photos*' -not -path "/mnt/wd-passport/immich/NO PEOPLE PHOTOS" -delete
|
||||
```
|
||||
|
||||
**Result from 3672 files**: all flattened in seconds. 1 filename collision (auto-renamed with `_1` suffix).
|
||||
|
||||
## Summary: Full Pipeline Run (16,187 Immich Photos)
|
||||
|
||||
```
|
||||
Initial classification (1-orientation, GPU): 3,954 → NO PEOPLE
|
||||
↓
|
||||
EXIF-aware restore (PIL + exif_transpose): 0 restored
|
||||
↓
|
||||
4-orientation brute-force restore: 280 restored
|
||||
↓
|
||||
Final in NO PEOPLE: 3,672 flattened into ALL Photos/
|
||||
```
|
||||
|
||||
**Total time**: ~70 min (47 min classify + 9 min EXIF restore + 9 min 4-orientation restore + ~5 min operations)
|
||||
|
||||
## Categorization Results (3,668 "No People" Photos)
|
||||
|
||||
After the full pipeline, a categorization pass was run on the 3,668 remaining photos using single-orientation YOLO to build a detection histogram:
|
||||
|
||||
**Result**: 2,743 photos had detections, 899 had no detections (blank/blurry/text-only screenshots), 26 errors (200MP panoramas).
|
||||
|
||||
**Top categories** — see the SKILL.md "Categorization by Detected Content" section for the full breakdown. The key takeaway: ~25% of sorted-out photos are blank/solid/blurry (no detections at all), and most of the detected ones are screenshots of phones/TVs/laptops or documents (books). Only ~15% are "maybe interesting" (cars, airplanes, food, clocks).
|
||||
- **HEIC photos** — `pip install pi-heif` needed or they silently fail (`PIL.UnidentifiedImageError: cannot identify image file`)
|
||||
- **Disk-bound** — sequential is fine, multiprocessing won't help if USB is the bottleneck
|
||||
- **Kill + resume** — no resume state is saved. Script must restart from scratch. For prod, add a SQLite checkpoint file
|
||||
- **GPU vs CPU** — `device="cuda:0"` requires PyTorch CUDA build. On Pascal GPUs (CC 6.x), PyTorch 2.x sometimes works with specific CUDA 12.x builds — test with `torch.cuda.is_available()` first
|
||||
- **"moved: X" in logs is queued, not actual** — The classify phase tracks `len(to_move)`, displayed as "moved: 2393". Actual file moves happen in a separate phase after classification completes. During classification, the destination directory stays empty.
|
||||
- **Immediate vs Batched file moves** — Two patterns. Batched (default): collect "to_move" list during classification, move in a separate phase. Log shows "moved: X" but destination is empty until the final phase. Immediate-move: `shutil.move()` inside the classification loop — destination fills in real-time, log counter is actual. Ask the user which they prefer. This configuration prefers **immediate moves**.
|
||||
- **Gateway restart kills agent-launched runs** — Even if launched with nohup, the process is a child of the agent's gateway session. Restarting the gateway kills it. Use `cronjob(action="create", no_agent=True, script="...")` for durable runs.
|
||||
- **Agent-launched runs waste API tokens** — The agent polls progress logs mid-turn, each poll = one LLM API call. Using `cronjob(no_agent=True)` avoids this entirely.
|
||||
|
||||
### Specific Error: `RuntimeError: GET was unable to find an engine to execute this computation`
|
||||
|
||||
**Cause**: PyTorch with CUDA 13.0 (cu130) on Pascal GPUs (CC 6.1). CUDA 13 dropped support for CC 6.x compute kernels. Memory management functions work (cudaMalloc etc.) but conv2d fails.
|
||||
|
||||
**Fix (modern)**: Use CUDA 12.6 wheels instead (forward-compatible with driver 580):
|
||||
```bash
|
||||
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu126
|
||||
```
|
||||
|
||||
**Fix (legacy)**: Python 3.10 + PyTorch 2.0.1+cu118 via uv (Python 3.10 + CUDA 11.8 build).
|
||||
|
||||
### Specific Error: `RuntimeError: Numpy is not available`
|
||||
|
||||
**Cause**: PyTorch 2.0.1 was compiled against numpy 1.x; `ultralytics` pulls in numpy 2.x.
|
||||
|
||||
**Fix**: Pin `numpy<2` after installing ultralytics.
|
||||
|
||||
### Specific Error: `RuntimeError: Error when binding input: There's no data transfer registered`
|
||||
|
||||
**Cause**: ONNX Runtime's CUDAExecutionProvider fails to load because `libcublasLt.so.12` is missing. The system has CUDA 13.0 installed, but onnxruntime-gpu was compiled against earlier CUDA.
|
||||
|
||||
**Fix**: Fall back to OpenVINO or use Option C (PyTorch CUDA 11.8).
|
||||
|
||||
### Specific Error: OOM (out of memory) with multiprocessing
|
||||
|
||||
**Symptom**: Workers start, process a few hundred images, then all die simultaneously. `dmesg` shows OOM killer activity.
|
||||
|
||||
**Cause**: `model(ap, verbose=False)` without `stream=True` accumulates all Results objects in RAM. Each worker holds results for all images it's processed so far. With 10 workers × ~200 images each = ~2000 full-resolution feature maps retained.
|
||||
|
||||
**Fix**: Pass `stream=True` to prevent accumulation, or reduce workers, or switch to sequential batch processing.
|
||||
|
||||
### Specific Warning: `WARNING ⚠️ NMS time limit 2.050s exceeded`
|
||||
|
||||
**Cause**: Non-maximum suppression (NMS) takes too long — typically because image decoding on a slow USB drive causes the pipeline to stall, then multiple images batch up in the worker's pipeline.
|
||||
|
||||
**Impact**: Self-correcting — NMS processes what it has and moves on. Not a hard error. Can indicate the USB drive is the bottleneck.
|
||||
|
||||
**Fix**: No action needed if results look correct. Consider reducing batch size or switching to sequential mode.
|
||||
@@ -0,0 +1,116 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Immich YOLO Photo Classifier Pipeline
|
||||
Classifies photos as: has-people, scenic, or neither.
|
||||
Moves non-people/non-scenic photos to NO PEOPLE PHOTOS folder.
|
||||
|
||||
Usage:
|
||||
/mnt/storage/yolo_venv/bin/python3 this_script.py
|
||||
|
||||
Requirements:
|
||||
- /mnt/storage/yolo_venv/ with ultralytics installed
|
||||
- ~/yolov8n_openvino_model/ (exported from yolov8n.pt)
|
||||
- Immich PostgreSQL container running
|
||||
- Photos on /mnt/wd-passport/immich/photos/
|
||||
"""
|
||||
import multiprocessing, os, time, warnings, subprocess, shutil
|
||||
warnings.filterwarnings("ignore")
|
||||
|
||||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||||
NO_PEOPLE_DIR = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
|
||||
DB_QUERY = '''docker exec immich_postgres psql -U postgres -d immich -t -A -c "SELECT \\\"originalPath\\\" FROM asset WHERE type='IMAGE' AND visibility='timeline' AND \\\"originalPath\\\" LIKE '/data/library/%' ORDER BY \\\"originalPath\\\";" 2>/dev/null'''
|
||||
|
||||
PERSON_CLASS = 0
|
||||
NATURE_CLASSES = {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 58, 77, 80}
|
||||
MOVE_CLASSES = {56, 57, 59, 60, 61, 62, 63, 64, 65, 67, 70, 71, 72, 73, 74, 75, 76, 78, 79,
|
||||
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}
|
||||
|
||||
def worker(args):
|
||||
model_path, paths = args
|
||||
from ultralytics import YOLO
|
||||
import warnings
|
||||
warnings.filterwarnings("ignore")
|
||||
model = YOLO(model_path, task="detect")
|
||||
results = []
|
||||
for ap in paths:
|
||||
try:
|
||||
r = model(ap, verbose=False)
|
||||
has_person = has_nature = has_move = False
|
||||
for r2 in r:
|
||||
if r2.boxes:
|
||||
for box in r2.boxes:
|
||||
cls = int(box.cls[0])
|
||||
if cls == PERSON_CLASS: has_person = True
|
||||
elif cls in NATURE_CLASSES: has_nature = True
|
||||
elif cls in MOVE_CLASSES: has_move = True
|
||||
if has_person: results.append((ap, "keep_people"))
|
||||
elif has_nature: results.append((ap, "keep_scenic"))
|
||||
elif has_move: results.append((ap, "move"))
|
||||
else: results.append((ap, "check_size"))
|
||||
except Exception as e:
|
||||
results.append((ap, f"error:{e}"))
|
||||
return results
|
||||
|
||||
if __name__ == "__main__":
|
||||
multiprocessing.set_start_method("forkserver", force=True)
|
||||
num_workers = min(multiprocessing.cpu_count(), 8)
|
||||
os.makedirs(NO_PEOPLE_DIR, exist_ok=True)
|
||||
model_path = os.path.expanduser("~/yolov8n_openvino_model/")
|
||||
|
||||
print(f"[1] Querying DB...", flush=True)
|
||||
result = subprocess.run(DB_QUERY, shell=True, capture_output=True, text=True, timeout=120)
|
||||
db_paths = [p.strip() for p in result.stdout.strip().split('\n') if p.strip()]
|
||||
|
||||
print(f"[2] Mapping {len(db_paths)} paths...", flush=True)
|
||||
actual = []
|
||||
for p in db_paths:
|
||||
ap = PHOTO_BASE + "/" + p[6:]
|
||||
if os.path.exists(ap):
|
||||
actual.append(ap)
|
||||
print(f" {len(actual)} accessible files", flush=True)
|
||||
|
||||
print(f"[3] Classifying with {num_workers} workers...", flush=True)
|
||||
chunks = [actual[i::num_workers] for i in range(num_workers)]
|
||||
chunk_args = [(model_path, c) for c in chunks]
|
||||
t_start = time.time()
|
||||
with multiprocessing.Pool(num_workers) as pool:
|
||||
all_results = pool.map(worker, chunk_args)
|
||||
elapsed = time.time() - t_start
|
||||
|
||||
keep_people = keep_scenic = errors = 0
|
||||
to_move, check_size = [], []
|
||||
for chunk in all_results:
|
||||
for ap, d in chunk:
|
||||
if d == "keep_people": keep_people += 1
|
||||
elif d == "keep_scenic": keep_scenic += 1
|
||||
elif d == "move": to_move.append(ap)
|
||||
elif d == "check_size": check_size.append(ap)
|
||||
else: errors += 1
|
||||
|
||||
print(f"\n YOLO: {elapsed/60:.1f}min ({len(actual)/elapsed:.1f} img/s)", flush=True)
|
||||
print(f" People:{keep_people} Scenic:{keep_scenic} Move:{len(to_move)} Check:{len(check_size)} Err:{errors}", flush=True)
|
||||
|
||||
print(f"[4] Size check on {len(check_size)} undetected...", flush=True)
|
||||
for ap in check_size:
|
||||
try:
|
||||
if os.path.getsize(ap) < 400000:
|
||||
to_move.append(ap)
|
||||
except: pass
|
||||
print(f" Now {len(to_move)} to move", flush=True)
|
||||
|
||||
print(f"[5] Moving {len(to_move)} photos...", flush=True)
|
||||
moved = file_errors = 0
|
||||
for ap in to_move:
|
||||
try:
|
||||
rel = os.path.relpath(ap, PHOTO_BASE)
|
||||
dest = os.path.join(NO_PEOPLE_DIR, rel)
|
||||
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||||
shutil.move(ap, dest)
|
||||
moved += 1
|
||||
except Exception as e:
|
||||
file_errors += 1
|
||||
|
||||
print(f"\nDONE! Moved {moved} photos", flush=True)
|
||||
print(f" Total:{len(actual)} Kept(people):{keep_people} Kept(scenic):{keep_scenic} Moved:{moved} Errors:{file_errors}", flush=True)
|
||||
print(f" Time: {(time.time()-t_start)/60:.1f}min", flush=True)
|
||||
print(f" Output: {NO_PEOPLE_DIR}", flush=True)
|
||||
@@ -0,0 +1,34 @@
|
||||
# Video Content Detection + Renaming — Production Script
|
||||
|
||||
A working YOLO-based pipeline for detecting video content from frames and renaming generically-named recovered video files (FILE000.MOV pattern).
|
||||
|
||||
## Script
|
||||
|
||||
Saved at `/tmp/rename_videos.py` and `/tmp/rename_videos2.py` on rayserver.
|
||||
|
||||
### Workflow
|
||||
1. List all files in the target directory starting with "FILE" (generically-named recovered files)
|
||||
2. Extract a video frame at 3 seconds using ffmpeg; fall back to 1 second for short clips
|
||||
3. Run YOLOv8n on CPU for content detection
|
||||
4. Build descriptive filename: `{people_count}_people_{object1}_{object2}.mov`
|
||||
5. Handle filename collisions with `_1`, `_2` suffixes
|
||||
6. Files with no detections → `scene.mov`
|
||||
|
||||
### Key design decisions
|
||||
- **CPU inference** — Viable for 80-100 files at ~3-5s/file. GPU not needed for this scale
|
||||
- **ffmpeg single-frame extraction** — Much faster than processing the entire video
|
||||
- **3-second offset** — Early enough to avoid blank intros, late enough to have content
|
||||
- **startswith("FILE") guard** — Makes the script idempotent (won't rename already-named files)
|
||||
|
||||
### Performance reference
|
||||
- **80 files**: ~4-6 minutes total (i7-10700K, CPU)
|
||||
- **~60 files** get descriptive names, **~20** remain as `scene` (blank/dark)
|
||||
|
||||
### Full script (80-file version)
|
||||
See `/tmp/rename_videos.py` and `/tmp/rename_videos2.py` for the two-phase version that handled 80 files.
|
||||
|
||||
### Edge cases
|
||||
- `.3GP` files are valid video containers — include them
|
||||
- Short clips (<3s): ffmpeg clips to available duration, produces a valid frame
|
||||
- Broken headers: ffmpeg hangs if file is truncated — always set `timeout=20`
|
||||
- Collisions: use `while os.path.exists(dst)` loop with incrementing counter
|
||||
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
GPU YOLO photo filter: keep people/scenic/screenshots, move everything else.
|
||||
Run with: python3 -u yolo_no_people_filter.py (-u for unbuffered output)
|
||||
|
||||
Customize SRC, DST, and class_sets below.
|
||||
|
||||
When running via Hermes background processes, ALWAYS use python3 -u and flush=True
|
||||
on all print() calls. Without this, progress output stays buffered and won't appear
|
||||
in the process viewer until the script exits.
|
||||
"""
|
||||
|
||||
import os, shutil
|
||||
from pathlib import Path
|
||||
from collections import Counter
|
||||
import torch
|
||||
from ultralytics import YOLO
|
||||
|
||||
# === CONFIGURATION ===
|
||||
SRC = "/mnt/seagate8tb/Photos" # Source directory (recursive)
|
||||
DST = "/mnt/seagate8tb/NO PEOPLE" # Destination for filtered photos
|
||||
BATCH_SIZE = 2 # 2 for 4GB GPU, 4 for 8GB+
|
||||
NUM_WORKERS = 4
|
||||
DEVICE = "cuda:0"
|
||||
CONFIDENCE = 0.3
|
||||
MODEL = "yolov8s.pt" # s=11M params (4GB safe), n=3M, m=26M
|
||||
HALF = True # FP16 for speed on CUDA
|
||||
|
||||
# === CLASSIFICATION SETS (COCO indices) ===
|
||||
PEOPLE_CLASSES = {0}
|
||||
|
||||
SCREENSHOT_CLASSES = {
|
||||
62, 63, 64, 65, 66, 67, # tv, laptop, mouse, remote, keyboard, cell phone
|
||||
73, 76, # book, scissors
|
||||
}
|
||||
|
||||
SCENIC_CLASSES = {
|
||||
1, 2, 3, 4, 6, 7, 8, # bicycle, car, motorcycle, airplane, bus, train, truck
|
||||
14, 15, 16, 17, 18, 19, # bird, cat, dog, horse, sheep, cow
|
||||
20, 21, 22, 23, # elephant, bear, zebra, giraffe
|
||||
25, 26, 27, 28, # umbrella, handbag, tie, suitcase
|
||||
29, 30, 31, 32, 33, 34, # frisbee, skis, snowboard, sports ball, kite, baseball bat
|
||||
35, 36, 37, 38, # baseball glove, skateboard, surfboard, tennis racket
|
||||
39, 40, 41, # bottle, wine glass, cup
|
||||
42, 43, 44, 45, 46, # fork, knife, spoon, bowl, banana
|
||||
47, 48, 49, 50, 51, # apple, sandwich, orange, broccoli, carrot
|
||||
52, 53, 54, 55, # hot dog, pizza, donut, cake
|
||||
56, 57, 58, 59, 60, # chair, couch, potted plant, bed, dining table
|
||||
72, 74, 75, 77, 78, 79, # clock, vase, teddy bear, hair drier, toothbrush
|
||||
80, 81, # hair drier, toothbrush (duplicates from COCO)
|
||||
}
|
||||
|
||||
|
||||
def is_scenic(detections):
|
||||
return any(cid in SCENIC_CLASSES for cid in detections)
|
||||
|
||||
|
||||
def is_screenshot(detections):
|
||||
return any(cid in SCREENSHOT_CLASSES for cid in detections)
|
||||
|
||||
|
||||
def main():
|
||||
print(f"Loading {MODEL} on {DEVICE}...", flush=True)
|
||||
model = YOLO(MODEL)
|
||||
model.to(DEVICE)
|
||||
|
||||
# Collect all image files
|
||||
extensions = {'.jpg', '.jpeg', '.png', '.webp', '.bmp',
|
||||
'.tiff', '.tif', '.gif', '.heic', '.heif'}
|
||||
files = []
|
||||
for root, dirs, fnames in os.walk(SRC):
|
||||
for f in fnames:
|
||||
if Path(f).suffix.lower() in extensions:
|
||||
files.append(os.path.join(root, f))
|
||||
|
||||
total = len(files)
|
||||
print(f"Found {total} images to scan", flush=True)
|
||||
|
||||
if total == 0:
|
||||
print("No images found!")
|
||||
return
|
||||
|
||||
stats = Counter()
|
||||
moved = 0
|
||||
kept_people = 0
|
||||
kept_scenic = 0
|
||||
kept_screenshot = 0
|
||||
errors = 0
|
||||
|
||||
for i in range(0, total, BATCH_SIZE):
|
||||
batch = files[i:i + BATCH_SIZE]
|
||||
batch_idx = i // BATCH_SIZE + 1
|
||||
total_batches = (total + BATCH_SIZE - 1) // BATCH_SIZE
|
||||
|
||||
try:
|
||||
results = model(batch, device=DEVICE, verbose=False,
|
||||
conf=CONFIDENCE, imgsz=640, half=HALF, stream=False)
|
||||
except Exception as e:
|
||||
# Batch failed — likely corrupt images in the mix.
|
||||
# Move them all directly to NO PEOPLE (no GPU retry — too slow).
|
||||
for filepath in batch:
|
||||
try:
|
||||
rel = os.path.relpath(filepath, SRC)
|
||||
dst = os.path.join(DST, rel)
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
shutil.move(filepath, dst)
|
||||
moved += 1
|
||||
stats['no_people'] += 1
|
||||
except Exception as e2:
|
||||
errors += 1
|
||||
continue
|
||||
|
||||
for result, filepath in zip(results, batch):
|
||||
boxes = result.boxes
|
||||
if boxes is None or len(boxes) == 0:
|
||||
# No detections → move
|
||||
rel = os.path.relpath(filepath, SRC)
|
||||
dst = os.path.join(DST, rel)
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
shutil.move(filepath, dst)
|
||||
moved += 1
|
||||
stats['empty'] += 1
|
||||
continue
|
||||
|
||||
class_ids = boxes.cls.cpu().numpy().astype(int)
|
||||
has_people = any(cid in PEOPLE_CLASSES for cid in class_ids)
|
||||
scenic = is_scenic(class_ids)
|
||||
screenshot = is_screenshot(class_ids)
|
||||
|
||||
if has_people:
|
||||
kept_people += 1
|
||||
stats['people'] += 1
|
||||
elif screenshot:
|
||||
kept_screenshot += 1
|
||||
stats['screenshot'] += 1
|
||||
elif scenic:
|
||||
kept_scenic += 1
|
||||
stats['scenic'] += 1
|
||||
else:
|
||||
rel = os.path.relpath(filepath, SRC)
|
||||
dst = os.path.join(DST, rel)
|
||||
os.makedirs(os.path.dirname(dst), exist_ok=True)
|
||||
try:
|
||||
shutil.move(filepath, dst)
|
||||
moved += 1
|
||||
stats['no_people'] += 1
|
||||
except Exception as e:
|
||||
print(f" Move error: {filepath}: {e}", flush=True)
|
||||
errors += 1
|
||||
|
||||
if batch_idx % 50 == 0 or batch_idx == total_batches:
|
||||
elapsed = min(batch_idx * BATCH_SIZE, total)
|
||||
pct = elapsed / total * 100
|
||||
print(f" [{elapsed}/{total}] {pct:.1f}% | moved={moved} | "
|
||||
f"people={kept_people} | scenic={kept_scenic} | "
|
||||
f"ss={kept_screenshot} | err={errors}", flush=True)
|
||||
|
||||
print(f"\n{'='*60}")
|
||||
print(f"DONE: {total} images processed")
|
||||
print(f" 👤 People (kept): {kept_people}")
|
||||
print(f" 🏔️ Scenic (kept): {kept_scenic}")
|
||||
print(f" 📱 Screenshot (kept): {kept_screenshot}")
|
||||
print(f" 📦 Moved to NO PEOPLE: {moved}")
|
||||
print(f" ❌ Errors: {errors}")
|
||||
print(f" 💾 Empty (no detections): {stats.get('empty', 0)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user