ray
850 lines
37 KiB
Markdown
850 lines
37 KiB
Markdown
---
|
||
name: computer-vision-inference
|
||
description: Run YOLO and other CV models on self-hosted servers for photo/video classification. Covers GPU compatibility checks, OpenVINO fallback, batch processing patterns, and integration with media servers (Immich).
|
||
---
|
||
|
||
# Computer Vision Inference on Self-Hosted Infrastructure
|
||
|
||
## User Preference: Plan Before Executing
|
||
|
||
When the user asks you to do a task, **stop and think** before running any command. Don't default to the first approach that comes to mind. Evaluate the most efficient way: consider the bottleneck (disk I/O, GPU, CPU), the right tools (dcraw, OpenVINO, CUDA), and whether the plan could fail. Getting the approach right upfront saves hours of rework. This applies to every task, not just CV — it's a general approach preference.
|
||
|
||
Run YOLO object detection on large photo collections (15k–50k+ images) on a homelab server.
|
||
|
||
## GPU Compatibility Check (First Step)
|
||
|
||
Before installing anything, verify the GPU can actually run modern ML frameworks:
|
||
|
||
```bash
|
||
# Check GPU and compute capability (CC)
|
||
nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader
|
||
# OR
|
||
nvidia-smi --query-gpu=name --format=csv,noheader
|
||
# Check CC at: https://developer.nvidia.com/cuda-gpus
|
||
```
|
||
|
||
**CC compatibility table:**
|
||
| Architecture | CC Range | PyTorch CUDA 12.x | CUDA 13.0+ | Notes |
|
||
|---|---|---|---|---|
|
||
| Turing (RTX 20xx) | 7.5 | ✅ Supported | ✅ Supported | Minimum for modern PyTorch |
|
||
| Ampere (RTX 30xx) | 8.x | ✅ Supported | ✅ Supported | |
|
||
| Ada Lovelace (RTX 40xx) | 8.9 | ✅ Supported | ✅ Supported | |
|
||
| **Pascal (GTX 10xx)** | **6.x** | ✅ Works (CUDA 12.4/12.6) | ❌ **Dropped** | sm_61 removed from CUDA 13.0 kernels |
|
||
|
||
**⚠️ Python 3.14 trap (Ubuntu 26.04):** The system Python 3.14 ships only PyTorch 2.12.0 wheels, which are compiled for CUDA 13.0. Pascal GPUs get `no kernel image is available` errors. **Fix:** install Python 3.12 from deadsnakes PPA and use CUDA 12.4 PyTorch wheels.
|
||
|
||
### YOLO Model Sizing by VRAM
|
||
|
||
| Model | Params | VRAM (batch=2, 640px) | Min GPU |
|
||
|-------|--------|----------------------|---------|
|
||
| YOLOv8n | 3.2M | ~400 MB | 2 GB |
|
||
| YOLOv8s | 11.2M | ~600 MB | 2 GB |
|
||
| YOLOv8m | 25.9M | ~1.2 GB | 3 GB |
|
||
| **YOLOv8l** | 43.7M | ~2 GB | 4 GB |
|
||
| **YOLOv8x** | 68.2M | ~3.5 GB | 6 GB |
|
||
|
||
**4GB card (GTX 1050 Ti / 1060):** Use YOLOv8s (safe) or YOLOv8m (tight but works). YOLOv8x will OOM at batch 2+. YOLOv8n is fastest but less accurate on small objects.
|
||
|
||
## Installation
|
||
|
||
### Option A: OpenVINO (for Intel CPUs or Pascal+ GPUs)
|
||
|
||
```bash
|
||
# Create venv on a partition with space (not /tmp)
|
||
sudo mkdir -p /mnt/storage/yolo_venv
|
||
sudo chown -R $USER:$USER /mnt/storage/yolo_venv
|
||
python3 -m venv /mnt/storage/yolo_venv
|
||
/mnt/storage/yolo_venv/bin/pip install ultralytics
|
||
|
||
# Export YOLO model to OpenVINO format
|
||
/mnt/storage/yolo_venv/bin/python3 -c "
|
||
from ultralytics import YOLO
|
||
model = YOLO('yolov8n.pt')
|
||
model.export(format='openvino', device='cpu')
|
||
"
|
||
# Saved as ~/yolov8n_openvino_model/
|
||
```
|
||
|
||
### Option B: PyTorch CUDA (for CC ≥ 7.5 GPUs)
|
||
|
||
```bash
|
||
python3 -m venv /mnt/storage/yolo_venv
|
||
/mnt/storage/yolo_venv/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
|
||
/mnt/storage/yolo_venv/bin/pip install ultralytics
|
||
```
|
||
|
||
### Option C: PyTorch CUDA for Pascal GPUs (CC 6.x -- GTX 1050 Ti, GTX 1060, etc.)
|
||
|
||
Pascal GPUs (CC 6.x) have no official PyTorch CUDA support in CUDA 13.0+. On Ubuntu 26.04 the system Python 3.14 ships ONLY PyTorch 2.12.0 with CUDA 13.0 — Pascal will fail with `no kernel image is available for execution on the device`.
|
||
|
||
**Approach 1: Python 3.12 + CUDA 12.4 (recommended for Ubuntu 26.04)**
|
||
|
||
Install Python 3.12 from deadsnakes PPA, create a venv, and use CUDA 12.4 PyTorch wheels:
|
||
|
||
```bash
|
||
# Install Python 3.12 (Ubuntu 26.04)
|
||
sudo add-apt-repository -y ppa:deadsnakes/ppa
|
||
sudo apt update && sudo apt install -y python3.12 python3.12-venv
|
||
|
||
# Create venv and install PyTorch CUDA 12.4
|
||
python3.12 -m venv /mnt/storage/yolo_venv_cu124
|
||
source /mnt/storage/yolo_venv_cu124/bin/activate
|
||
pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 \
|
||
--index-url https://download.pytorch.org/whl/cu124
|
||
pip install ultralytics pillow numpy
|
||
|
||
# Verify
|
||
python3 -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')"
|
||
```
|
||
|
||
**Approach 2: Python 3.10 + PyTorch 2.0.1 + CUDA 11.8 (legacy stable)**
|
||
|
||
|
||
```bash
|
||
# 1. Install uv (Python 3.14+ system with PEP 668? uv bypasses it)
|
||
pip3 install uv 2>&1 || /mnt/storage/yolo_venv/bin/pip install uv
|
||
|
||
# 2. Install Python 3.10
|
||
export PATH="$HOME/.local/bin:$PATH"
|
||
uv python install 3.10
|
||
|
||
# 3. Create venv with Python 3.10
|
||
uv venv --python 3.10 /path/to/gpu_venv
|
||
|
||
# 4. Install PyTorch 2.0.1+cu118 (supports CUDA 11.8 + CC 6.1)
|
||
uv pip install --python /path/to/gpu_venv/bin/python \
|
||
torch==2.0.1+cu118 torchvision==0.15.2+cu118 \
|
||
--index-url https://download.pytorch.org/whl/cu118
|
||
|
||
# 5. Pin numpy < 2 (PyTorch 2.0.1 was compiled against numpy 1.x)
|
||
uv pip install --python /path/to/gpu_venv/bin/python "numpy<2"
|
||
|
||
# 6. Install ultralytics
|
||
uv pip install --python /path/to/gpu_venv/bin/python ultralytics
|
||
# If pi-heif fails via uv, install pip first then use pip:
|
||
uv pip install --python /path/to/gpu_venv/bin/python pip
|
||
/path/to/gpu_venv/bin/pip install pi-heif
|
||
|
||
# 7. Verify GPU
|
||
/path/to/gpu_venv/bin/python3 -c "
|
||
import torch
|
||
print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')
|
||
t = torch.zeros(1,3,640,640).cuda()
|
||
conv = torch.nn.Conv2d(3, 16, 3).cuda()
|
||
print(f'CUDA compute: OK ({conv(t).shape})')
|
||
"
|
||
```
|
||
|
||
**Performance reference (GTX 1050 Ti, USB HDD):** ~4–6 img/s at batch=4. Bottleneck is disk I/O, not GPU.
|
||
|
||
## YOLO Inference Patterns
|
||
|
||
Templates available:
|
||
- `templates/yolo_no_people_filter.py` — GPU batch scan: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).
|
||
|
||
### Single image (test):
|
||
|
||
```python
|
||
from ultralytics import YOLO
|
||
model = YOLO("yolov8n_openvino_model/")
|
||
results = model("/path/to/photo.jpg", verbose=False)
|
||
for r in results:
|
||
if r.boxes:
|
||
for box in r.boxes:
|
||
cls = int(box.cls[0])
|
||
name = r.names[cls]
|
||
conf = float(box.conf[0])
|
||
print(f" {name}: {conf:.2f}")
|
||
```
|
||
|
||
### EXIF-aware inference (fixes sideways photos with orientation flags)
|
||
|
||
YOLO loads images via OpenCV (`cv2.imread`) which **ignores EXIF orientation flags**. A portrait photo taken with a phone appears rotated 90° to the model, causing missed person/object detections. Fix by pre-rotating with PIL. **However, this only catches photos that HAVE EXIF orientation flags — some photos are stored with rotated pixel data and no orientation flag at all.** See the section below for the brute-force fallback.
|
||
|
||
```python
|
||
from PIL import Image, ImageOps
|
||
import numpy as np
|
||
|
||
def load_image_exif_aware(path: str) -> np.ndarray:
|
||
"""Load image with EXIF rotation applied, return BGR numpy array.
|
||
|
||
Note: ImageOps.exif_transpose returns None if the image has
|
||
no EXIF orientation tag. Always handle None by reloading fresh.
|
||
"""
|
||
with Image.open(path) as img:
|
||
img = ImageOps.exif_transpose(img)
|
||
if img is None:
|
||
# No EXIF orientation data — reload fresh
|
||
return None # Caller should fall back to brute-force rotation
|
||
if img.mode == "RGBA":
|
||
img = img.convert("RGB")
|
||
elif img.mode != "RGB":
|
||
img = img.convert("RGB")
|
||
return np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||
|
||
img_np = load_image_exif_aware("/path/to/photo.jpg")
|
||
if img_np is not None:
|
||
results = model(img_np, verbose=False)
|
||
```
|
||
|
||
**Performance**: PIL load + EXIF transpose is fast (~11 img/s on GTX 1050 Ti) — the rotate is a metadata operation, not a pixel resample.
|
||
|
||
## Multi-Orientation Detection (Critical for Rotated Photos)
|
||
|
||
YOLO's default pipeline (OpenCV `cv2.imread`) ignores EXIF orientation flags and stores pixels as-is. A portrait phone photo appears 90° rotated to the model, causing missed detections. Fixing orientation uncovered **280+ missed people photos** in a 4,000-photo corpus.
|
||
|
||
### Two-Layer Strategy
|
||
|
||
Layer 1: EXIF-aware PIL loading (fast, catches photos with orientation flags)
|
||
Layer 2: 4-orientation brute force (definitive, catches photos stored rotated without EXIF flags)
|
||
|
||
| Layer | Speed | Found on 4k photos | Time |
|
||
|-------|-------|-------------------|------|
|
||
| EXIF-only (PIL + `ImageOps.exif_transpose`) | ~11 img/s | **0** missed photos | ~6 min |
|
||
| 4-orientation brute force | ~4.5 img/s | **280+** missed photos | ~25 min |
|
||
|
||
**Key takeaway:** EXIF-only found nothing. 4-orientation found massive numbers of missed people photos. Many phones and apps (especially older Android, TikTok screenshots, and re-saved images) store rotated pixel data **without** setting the EXIF orientation flag. EXIF handling alone is insufficient.
|
||
|
||
### Helper: try_all_orientations()
|
||
|
||
The definitive approach: rotate the image array 4 ways and check each for people (class 0).
|
||
|
||
```python
|
||
import numpy as np
|
||
from PIL import Image
|
||
|
||
def try_all_orientations(ap: str, model) -> tuple[bool, str]:
|
||
"""Check all 4 rotations of a photo for people. Returns (found_person, orientation_name)."""
|
||
try:
|
||
with Image.open(ap) as img:
|
||
if img.mode != 'RGB':
|
||
img = img.convert('RGB')
|
||
img_np = np.array(img)[:, :, ::-1] # RGB -> BGR for YOLO
|
||
except Exception:
|
||
return (False, "")
|
||
|
||
orientations = {
|
||
'0°': img_np,
|
||
'90°': np.rot90(img_np, k=3), # 90° CW
|
||
'180°': np.rot90(img_np, k=2), # 180°
|
||
'270°': np.rot90(img_np, k=1), # 270° CW
|
||
}
|
||
|
||
for orient_name, orient_img in orientations.items():
|
||
try:
|
||
results = model(orient_img, device="cuda:0", verbose=False)
|
||
except:
|
||
continue
|
||
for r in results:
|
||
if r.boxes:
|
||
for box in r.boxes:
|
||
if int(box.cls[0]) == 0: # person
|
||
return (True, orient_name)
|
||
return (False, "")
|
||
|
||
# Usage in restore loop:
|
||
found, orientation = try_all_orientations(ap, model)
|
||
if found:
|
||
print(f"Found person at {orientation}")
|
||
shutil.move(ap, original_path)
|
||
```
|
||
|
||
**Performance**: ~4.5 img/s (4× slower than single-orientation). ~25 min for 4,000 photos. RAM stays stable (tested on GTX 1050 Ti).
|
||
|
||
### Full dual-pass workflow
|
||
|
||
```python
|
||
# Pass 1: EXIF-aware (fast pre-filter)
|
||
for ap in moved_photos:
|
||
img_bgr = load_with_exif(ap) # tries EXIF transpose
|
||
if img_bgr is None:
|
||
continue # nothing to see RAW — pass to brute force
|
||
results = model(img_bgr, verbose=False)
|
||
if has_person(results):
|
||
restore(ap)
|
||
|
||
# Pass 2: 4-orientation (catches everything else)
|
||
for ap in still_in_moved_photos:
|
||
found, orient = try_all_orientations(ap, model)
|
||
if found:
|
||
restore(ap)
|
||
```
|
||
|
||
**Practical note**: Do pass 2 even if pass 1 found nothing — on a real corpus, pass 1 found 0 but pass 2 found 280+. The EXIF orientation flag is simply not present in many stored photos.
|
||
|
||
### Edge cases when loading images via PIL
|
||
|
||
| Problem | Cause | Fix |
|
||
|---------|-------|-----|
|
||
| RGBA PNG | PNG screenshots with alpha channel produce 4-channel arrays | `img = img.convert("RGB")` |
|
||
| CMYK/grayscale | Some scanned images or legacy formats | `img = img.convert("RGB")` |
|
||
| Decompression bomb (200MP+ panoramas) | Pillow error: "Image size exceeds limit" | `PIL.Image.MAX_IMAGE_PIXELS = None` or catch `PIL.DecompressionBombError` and skip; the latter is safer |
|
||
| HEIC/HEIF | iPhone photos | `pip install pi-heif` |
|
||
| Nikon RAW (NEF) | DSLR RAW files from Nikon cameras — 20-24MB each, 6000x4000px | Install `dcraw` (`sudo apt install dcraw`) for command-line conversion. Use `dcraw -c -w <file.NEF>` to pipe decoded PPM to stdout. Do NOT use rawpy/LibRaw — it fails on many camera-specific NEF variants (especially Nikon D5200+ and other consumer bodies). dcraw is the gold standard. Example: `dcraw -c -w photo.NEF | convert - photo.jpg` |
|
||
| Truncated JPEG | Corrupt or partial downloads | Wrap in try/except, skip on error |
|
||
| Missing EXIF orientation flag | Photo pixels are rotated but no EXIF flag set | Use 4-orientation brute force detection (see multi-orientation section) |
|
||
|
||
### Immediate-move pattern (files moved as classified)
|
||
|
||
When the user wants photos moved/purged as they're processed (not batched at the end), move inside the classification loop:
|
||
|
||
```python
|
||
moved_total = 0
|
||
for i, ap in enumerate(paths):
|
||
results = model(ap, device="cuda:0", verbose=False)
|
||
|
||
should_move = False
|
||
# ... classify results ...
|
||
|
||
if should_move:
|
||
try:
|
||
dest = ap.replace(PHOTO_DIR, NO_PEOPLE_DIR)
|
||
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||
shutil.move(ap, dest)
|
||
moved_total += 1
|
||
except Exception as e:
|
||
log(f"MOVE FAILED: {ap} -> {e}")
|
||
```
|
||
|
||
**Important**: Unlike the batched pattern where "moved: X" means "queued for move", in immediate mode `moved_total` is the actual count of files already in the destination folder.
|
||
|
||
### Multiprocessing batch (use `if __name__ == "__main__"` guard):
|
||
|
||
```python
|
||
import multiprocessing
|
||
|
||
def worker(args):
|
||
model_path, paths = args
|
||
from ultralytics import YOLO
|
||
import warnings
|
||
warnings.filterwarnings("ignore")
|
||
model = YOLO(model_path, task="detect")
|
||
results = []
|
||
for ap in paths:
|
||
r = model(ap, verbose=False)
|
||
results.append((ap, decision))
|
||
return results
|
||
|
||
if __name__ == "__main__":
|
||
multiprocessing.set_start_method("forkserver", force=True)
|
||
num_workers = min(multiprocessing.cpu_count(), 8)
|
||
chunks = [all_paths[i::num_workers] for i in range(num_workers)]
|
||
chunk_args = [(model_path, c) for c in chunks]
|
||
with multiprocessing.Pool(num_workers) as pool:
|
||
all_results = pool.map(worker, chunk_args)
|
||
```
|
||
|
||
## COCO Class Classification Heuristic
|
||
|
||
```python
|
||
PERSON_CLASS = 0
|
||
NATURE_CLASSES = {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 58, 77, 80}
|
||
MOVE_CLASSES = {56, 57, 59, 60, 61, 62, 63, 64, 65, 67, 70, 71, 72, 73, 74, 75, 76, 78, 79,
|
||
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}
|
||
```
|
||
|
||
Rules: Person→keep, Nature→keep, Indoor/urban→move, No detections→check file size (<400KB→move)
|
||
|
||
## Integration with Immich
|
||
|
||
### Query asset paths from PostgreSQL:
|
||
|
||
```python
|
||
DB_QUERY = '''docker exec immich_postgres psql -U postgres -d immich -t -A -c "SELECT \\\"originalPath\\\" FROM asset WHERE type='IMAGE' AND visibility='timeline' AND \\\"originalPath\\\" LIKE '/data/library/%' ORDER BY \\\"originalPath\\\";" 2>/dev/null'''
|
||
```
|
||
|
||
### Map virtual paths to filesystem:
|
||
|
||
```python
|
||
PHOTO_BASE = "/mnt/wd-passport/immich/photos"
|
||
def map_path(db_path):
|
||
if db_path.startswith("/data/"):
|
||
return PHOTO_BASE + "/" + db_path[6:]
|
||
elif db_path.startswith("/external/"):
|
||
return "/mnt/wd-passport" + db_path[8:]
|
||
return db_path
|
||
```
|
||
|
||
### Flatten sorted photos into a single folder
|
||
|
||
After classification, photos end up scattered across Immich's subdirectory tree (e.g., `NO PEOPLE PHOTOS/library/4defcb72-.../2026/2026-04-07/photo.jpg`). To flatten them into a single folder with no subdirectories:
|
||
|
||
```python
|
||
import os, shutil
|
||
|
||
NO_PEOPLE = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
|
||
TARGET = os.path.join(NO_PEOPLE, "ALL Photos")
|
||
os.makedirs(TARGET, exist_ok=True)
|
||
|
||
# Find all files excluding the target itself
|
||
files = []
|
||
for root, dirs, fnames in os.walk(NO_PEOPLE):
|
||
if root.startswith(TARGET):
|
||
continue
|
||
for f in fnames:
|
||
files.append(os.path.join(root, f))
|
||
|
||
moved = 0
|
||
collisions = 0
|
||
for src in files:
|
||
fname = os.path.basename(src)
|
||
dst = os.path.join(TARGET, fname)
|
||
if os.path.exists(dst):
|
||
base, ext = os.path.splitext(fname)
|
||
n = 1
|
||
while os.path.exists(os.path.join(TARGET, f"{base}_{n}{ext}")):
|
||
n += 1
|
||
dst = os.path.join(TARGET, f"{base}_{n}{ext}")
|
||
collisions += 1
|
||
shutil.move(src, dst)
|
||
moved += 1
|
||
|
||
print(f"Moved {moved} files, {collisions} renamed for collision")
|
||
|
||
# Clean up empty directories
|
||
for root, dirs, fnames in os.walk(NO_PEOPLE, topdown=False):
|
||
if root != TARGET and not os.listdir(root) and root != NO_PEOPLE:
|
||
os.rmdir(root)
|
||
```
|
||
|
||
**Expectation**: 3,600+ files flatten in seconds. Filename collisions are rare (~1 in 3,600). Collision handling appends `_1`, `_2`, etc. before the extension.
|
||
|
||
## Categorization by Detected Content
|
||
|
||
After sorting photos out (no people, no scenery), you may want to further categorize the remaining photos by what objects they actually contain. Run YOLO on every photo and build a detection histogram, then let the user decide which categories are worthwhile and which are trash.
|
||
|
||
### Detection Histogram Script
|
||
|
||
```python
|
||
import os, numpy as np
|
||
from PIL import Image
|
||
from collections import Counter
|
||
from ultralytics import YOLO
|
||
|
||
ALL_PHOTOS = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
|
||
model = YOLO("yolov8n.pt")
|
||
|
||
# Gather all image files
|
||
files = sorted([os.path.join(ALL_PHOTOS, f) for f in os.listdir(ALL_PHOTOS)
|
||
if f.lower().endswith(('.jpg', '.jpeg', '.png', '.heic', '.heif'))])
|
||
|
||
detection_counts = Counter() # class_id -> count of photos containing it
|
||
no_detect = 0
|
||
|
||
for i, ap in enumerate(files):
|
||
if i > 0 and i % 300 == 0:
|
||
print(f" [{i}/{len(files)}] ... no_detect: {no_detect}", flush=True)
|
||
|
||
# Load with PIL (handles EXIF + RGBA)
|
||
with Image.open(ap) as img:
|
||
if img.mode != 'RGB':
|
||
img = img.convert('RGB')
|
||
img_np = np.array(img)[:, :, ::-1]
|
||
|
||
results = model(img_np, device="cuda:0", verbose=False)
|
||
|
||
classes_found = set()
|
||
for r in results:
|
||
if r.boxes:
|
||
for box in r.boxes:
|
||
classes_found.add(int(box.cls[0]))
|
||
|
||
if classes_found:
|
||
for c in classes_found:
|
||
detection_counts[c] += 1
|
||
else:
|
||
no_detect += 1
|
||
|
||
# Print top results
|
||
CLASS_NAMES = {0:"person",1:"bicycle",2:"car",3:"motorcycle",15:"cat",16:"dog",
|
||
39:"bottle",41:"cup",56:"chair",62:"tv",63:"laptop",67:"cell phone",
|
||
73:"book",74:"clock"} # add more as needed
|
||
for cls_id, count in detection_counts.most_common(20):
|
||
name = CLASS_NAMES.get(cls_id, f"c{cls_id}")
|
||
print(f" {name:20s}: {count:4d} photos ({count/len(files)*100:.1f}%)")
|
||
print(f" {'no detection':20s}: {no_detect:4d} photos ({no_detect/len(files)*100:.1f}%)")
|
||
```
|
||
|
||
### Real-World Detection Distribution (3,668 photos sorted out as "no people/scenery")
|
||
|
||
| Object | Photos | % | Likely |
|
||
|--------|--------|---|--------|
|
||
| book | 668 | 18% | Documents, receipts, textbooks |
|
||
| cell phone | 342 | 9% | Phone screenshots |
|
||
| tv/monitor | 320 | 9% | Screen captures |
|
||
| laptop | 274 | 7% | Computer screenshots |
|
||
| car | 265 | 7% | Vehicle photos |
|
||
| refrigerator | 218 | 6% | Fridge contents |
|
||
| clock | 128 | 3% | Timestamps |
|
||
| chair | 102 | 3% | Random furniture |
|
||
| dining table | 74 | 2% | Meals |
|
||
| toilet | 74 | 2% | Bathroom shots |
|
||
| airplane | 54 | 1.5% | Travel |
|
||
| No detection | 899 | 25% | Blank/blurry/text-only |
|
||
|
||
### Sorting Into Subfolders by Category
|
||
|
||
After the histogram gives you a picture of what's there, present it to the user and ask which categories to keep vs discard. Then sort:
|
||
|
||
```python
|
||
import os, shutil
|
||
|
||
TRASH_CLASSES = {62, 63, 67, 73} # tv, laptop, cell phone, book
|
||
TRASH_DIR = os.path.join(ALL_PHOTOS, "..", "Trash Screenshots")
|
||
os.makedirs(TRASH_DIR, exist_ok=True)
|
||
|
||
KEEP_CLASSES = {2, 5, 4, 39, 41, 74} # car, bus, airplane, bottle, cup, clock
|
||
KEEP_DIR = os.path.join(ALL_PHOTOS, "..", "Maybe Keep")
|
||
os.makedirs(KEEP_DIR, exist_ok=True)
|
||
|
||
for fname in os.listdir(ALL_PHOTOS):
|
||
ap = os.path.join(ALL_PHOTOS, fname)
|
||
if not os.path.isfile(ap):
|
||
continue
|
||
# ... check fname against a pre-computed CSV of classes ...
|
||
shutil.move(ap, dest)
|
||
```
|
||
|
||
**Practical tip**: Instead of re-running YOLO on every photo, save the classification results to a CSV during the detection histogram pass. Then use the CSV to sort — much faster:
|
||
|
||
```python
|
||
# During detection pass, write:
|
||
# all_photo_categories.csv
|
||
# filename,detected_classes
|
||
with open("all_photo_categories.csv", "w") as f:
|
||
f.write("filename,detected_classes\n")
|
||
for fname, classes in photo_classes.items():
|
||
names = ";".join(sorted(CLASS_NAMES.get(c, f"c{c}") for c in classes))
|
||
f.write(f"{fname},{names}\n")
|
||
# Also record no-detection photos
|
||
for fname in no_detect_files:
|
||
f.write(f"{fname},none\n")
|
||
```
|
||
|
||
Then sort using the CSV — instant classification lookup, no GPU needed.
|
||
|
||
## Pitfalls
|
||
|
||
- **HEIC photos**: `pip install pi-heif` for HEIC/HEIF support
|
||
- **Slow USB drives**: Use PostgreSQL DB query instead of filesystem `find`/`walk` on external drives with 20k+ files
|
||
- **Docker bridge IP loss**: If bridge lacks gateway IP, do full `docker compose down && docker compose up -d`
|
||
- **OpenVINO batching**: LATENCY mode won't batch — use multiprocessing for parallelism
|
||
- **Venv placement**: PyTorch+ultralytics is ~2-3GB. Install on a data partition, not `/tmp`
|
||
- **Multiprocessing**: Must use `if __name__ == "__main__"` guard with `set_start_method("forkserver")` in Python 3.14+
|
||
- **`stream=True` to prevent OOM**: Without `stream=True`, YOLO holds all results in RAM across the entire run. For 15k+ images, each result with boxes metadata accumulates until the process is killed by OOM killer. Always pass `stream=True` to `model(ap, verbose=False, stream=True)` when iterating through many images. Note: `stream=True` returns a generator; materialize it with `list()` or iterate immediately.
|
||
- **Zombie GPU processes**: Killing a YOLO script (Ctrl+C, `kill`, or OOM crash) often leaves a zombie Python process holding all VRAM. The GPU shows 4024MiB / 4096MiB used but no visible process. Run `nvidia-smi` to find the PID, then `kill -9 <pid>`. Always check `nvidia-smi` after a failed run — the GPU won't be usable until the zombie is cleared.
|
||
- **Cascading CUDA failures after zombie**: If a new YOLO run immediately OOMs on a model that previously fit (e.g. YOLOv8s on 4GB), check `nvidia-smi` for a ghost process from a prior run. The "out of memory" message is correct — the GPU IS full, just by a process that appears dead.
|
||
- **Corrupt file batch fallback trap**: When a YOLO batch fails due to corrupt images, the naive fallback re-tries each image individually on GPU at reduced resolution. On a 4GB card with dozens of corrupt files (common in recovery data), this can stall the pipeline for hours — each retry re-loads the model, allocates CUDA context, and fails. **Fix**: treat batch-level failures as a corrupt batch. Move all images in the failed batch directly to the destination without any GPU retry. If the user asks to classify corrupt images separately, use PIL to check for readability first (fast CPU operation), then only run YOLO on readable files.
|
||
- **Background output buffering via SSH**: When launching a long-running Python script via `terminal(background=true)` over SSH, standard output is often completely invisible — the SSH pipe buffers aggressively and the process viewer shows empty output for minutes. **Fix**: redirect stdout+stderr to a log file on the remote server (`> /tmp/script.log 2>&1`), then monitor the log file with `read_file()`. Also use `python3 -u` (unbuffered mode) and `flush=True` on all print() calls.
|
||
- **NumPy version mismatch**: PyTorch 2.0.1 was compiled against numpy 1.x. Installing ultralytics pulls in numpy 2.x, causing `RuntimeError: Numpy is not available` at inference time. Pin `numpy<2` in the venv after installing everything else.
|
||
- **`setsid` for background isolation**: `nohup` is insufficient for SSH-launched background processes — `setsid` detaches from the terminal session entirely, preventing SIGHUP from the SSH closure. Use: `setsid python /tmp/script.py > /home/ray/log 2>&1 &`
|
||
|
||
## GPU Memory FAQ
|
||
|
||
Users frequently ask "why is my GPU only using X MB when it has Y GB available?" during YOLO inference:
|
||
|
||
### Why VRAM usage stays low (e.g. 432MB / 4GB)
|
||
|
||
| Factor | Why |
|
||
|--------|-----|
|
||
| **Model size** | YOLOv8n (nano) is ~6.3MB. Weights + CUDA overhead = ~200–500MB VRAM. YOLOv8x would use ~1–2GB |
|
||
| **Batch size = 1** | Each frame is processed and discarded before the next. No gradient history, no activation cache (those are training-only) |
|
||
| **Inference ≠ Training** | Training batches many images, backpropagates through the full graph, and caches activations for gradient computation — that's what fills VRAM. Inference is forward-pass-only |
|
||
| **CUDA context floor** | PyTorch reserves ~300–500MB of VRAM just for the CUDA runtime context regardless of model size |
|
||
|
||
The 1050 Ti's 4GB is sitting idle because the workload doesn't need more — not because something's misconfigured. YOLOv8n at batch-1 on a single image simply doesn't fill a 4GB buffer.
|
||
|
||
### Why system RAM is high even with GPU inference
|
||
|
||
If a YOLO process shows 3.7GB RSS in `ps aux` but only 432MB in `nvidia-smi`:
|
||
|
||
| Layer | RAM |
|
||
|-------|-----|
|
||
| Python + PyTorch + Ultralytics + CUDA libs loaded in memory | ~1.5 GB |
|
||
| Image decode buffers (reads JPEGs from disk into pixel arrays) | varies |
|
||
| PyTorch pre-allocated CPU tensor arena | ~500 MB |
|
||
| CUDA driver / runtime bookkeeping on CPU side | ~200 MB |
|
||
| Python overhead (file path list, results dicts for 15k+ images) | ~200 MB |
|
||
|
||
The GPU is the calculator, but the CPU is the assistant that reads files from disk, decodes JPEGs, normalizes pixels, and copies tensors over. The assistant needs a desk (RAM) to work at.
|
||
|
||
### Sequential Batch Processing (for disk-bound pipelines)
|
||
|
||
## Comparing NEF (RAW) vs JPG Pairs with YOLO
|
||
|
||
Useful when you need to verify that paired RAW+JPEG files (same base filename in a Nikon camera's dual-save setup) are actually the same photo. The JPEG is the camera's processed output, the NEF is the sensor data — they should be the same image with different processing.
|
||
|
||
### Pitfall: rawpy/LibRaw fails on many consumer NEFs
|
||
|
||
`rawpy` (Python binding for LibRaw) produces "data corrupted" errors on many Nikon D5200 and similar consumer-body NEF files. These errors are **false positives** — the files open fine in Lightroom, Nikon's software, and `dcraw`. Always fall back to dcraw for NEF conversion.
|
||
|
||
### dcraw + YOLO comparison workflow
|
||
|
||
```python
|
||
import subprocess, io
|
||
from PIL import Image
|
||
|
||
# Convert NEF to PPM via dcraw (piped to stdout)
|
||
r = subprocess.run(['dcraw', '-c', '-w', str(nef_path)],
|
||
capture_output=True, timeout=60)
|
||
if r.returncode != 0:
|
||
raise Exception(f"dcraw failed: {r.stderr.decode()[:80]}")
|
||
|
||
# Read PPM from stdout
|
||
nef_img = Image.open(io.BytesIO(r.stdout)).convert('RGB')
|
||
jpg_img = Image.open(jpg_path).convert('RGB')
|
||
|
||
# Resize both to same dimensions for comparison
|
||
target = (640, 480)
|
||
nef_r = nef_img.resize(target, Image.LANCZOS)
|
||
jpg_r = jpg_img.resize(target, Image.LANCZOS)
|
||
|
||
# Run YOLO on CPU
|
||
from ultralytics import YOLO
|
||
model = YOLO('yolov8n.pt')
|
||
jr = model(jpg_r, device='cpu', verbose=False)
|
||
nr = model(nef_r, device='cpu', verbose=False)
|
||
|
||
# Compare detected classes
|
||
jc = sorted([model.names[int(b.cls)] for b in jr[0].boxes])
|
||
nc = sorted([model.names[int(b.cls)] for b in nr[0].boxes])
|
||
|
||
if jc == nc:
|
||
print("Same photo — identical YOLO detections")
|
||
else:
|
||
print(f"Different detections: JPG={jc} vs NEF={nc}")
|
||
```
|
||
|
||
### Expected results
|
||
|
||
The NEF version typically detects **more objects** than the JPEG version because it has the full sensor dynamic range (24MP, deeper shadows/highlights). A JPG that sees 4 people might detect 8 in the NEF, all in shadow areas the JPG crushed. This **confirms they're the same photo** with richer data in the RAW version — not different photos.
|
||
|
||
### NEF size reference
|
||
|
||
| Camera | NEF size | JPEG size | Resolution |
|
||
|--------|----------|-----------|------------|
|
||
| Nikon D5200 | ~23 MB | ~2-8 MB | 6036 × 4020 |
|
||
|
||
### When to use CPU vs GPU for verification
|
||
|
||
If the installed PyTorch version doesn't support the GPU's compute capability (common with GTX 1050 Ti / CC 6.1 on modern PyTorch CUDA 12+), force CPU:
|
||
```python
|
||
model(image, device='cpu', verbose=False)
|
||
```
|
||
CPU inference on 640×480 images with YOLOv8n is ~1-2 img/s per core. For 100 images, expect ~2-3 minutes — acceptable for verification tasks.
|
||
|
||
When the bottleneck is disk I/O (reading from a USB HDD), not GPU compute, multiprocessing doesn't help much. A simple sequential loop is simpler and more predictable:
|
||
|
||
```python
|
||
from ultralytics import YOLO
|
||
model = YOLO("yolov8n.pt")
|
||
|
||
counts = defaultdict(int)
|
||
to_move = []
|
||
t_start = time.time()
|
||
|
||
for i, ap in enumerate(all_paths):
|
||
if i > 0 and i % 500 == 0:
|
||
elapsed = time.time() - t_start
|
||
rate = i / elapsed
|
||
eta = (len(all_paths) - i) / rate / 60
|
||
print(f" [{i}/{len(all_paths)}] {rate:.1f} img/s, ETA {eta:.0f}min", flush=True)
|
||
|
||
try:
|
||
results = model(ap, device="cuda:0", verbose=False)
|
||
except:
|
||
continue
|
||
|
||
# classify results...
|
||
```
|
||
|
||
**Performance reference (GTX 1050 Ti, USB HDD):** ~6–7 img/s on 16k images, ~40 min total for 16k photos.
|
||
|
||
## Running Long Inference Without Blocking the Hermes Gateway
|
||
|
||
**Critical:** If the agent launches a long classification script inside its conversation turn, the Telegram/Discord bot appears "typing..." and **cannot process new messages** until the turn finishes. Each agent iteration also makes an LLM API call — polling progress wastes tokens while the user is charged per-call.
|
||
|
||
### ❌ Anti-pattern: Agent launches script and waits
|
||
|
||
```
|
||
User: "classify my photos"
|
||
Agent: runs terminal("python classify.py") ← blocks for 40 min
|
||
User: "what's the status?" ← queued, not seen
|
||
...bot shows typing for 40 min...
|
||
← agent finally responds (dozens of API calls eaten)
|
||
```
|
||
|
||
### ✅ Pattern A: Durable cron job (recommended)
|
||
|
||
Create a cron job with `no_agent=True` — the script runs independently, stdout goes directly to Telegram. Survives gateway restart, zero token cost during execution:
|
||
|
||
```
|
||
cronjob(
|
||
action="create",
|
||
schedule="now",
|
||
script="/home/ray/yolo_classify.py",
|
||
no_agent=True,
|
||
deliver="origin"
|
||
)
|
||
```
|
||
|
||
### ✅ Pattern B: terminal(background=true) + log file + end turn
|
||
|
||
Launch with `background=true`, redirect stdout to a log file (SSH pipe buffers aggressively — without this, output is invisible), and immediately end the turn. Bot becomes responsive but **does not survive gateway restart** — child processes are killed when the gateway restarts:
|
||
|
||
```bash
|
||
# Redirect to a log file to avoid SSH pipe buffering
|
||
terminal(command="bash -c 'source venv/bin/activate && python3 -u classify.py > /tmp/classify.log 2>&1'", background=true, notify_on_complete=true)
|
||
# Monitor progress via read_file("/tmp/classify.log")
|
||
```
|
||
|
||
**Note:** Always use `python3 -u` (unbuffered) and `flush=True` on print() calls, even with log redirection. Without `-u`, Python's own output buffering delays log writes and `read_file()` shows stale data.
|
||
|
||
**This user's preference**: Always run long tasks in the background. They want to stay able to message me while work is processing — no waiting for 10-40 min tasks to finish before I respond.
|
||
```
|
||
|
||
### Pitfall: "moved: X" is queued, not actually moved
|
||
|
||
In a 2-phase script (classify all → then move all), the log shows `moved: 2393` during classification. This is `len(to_move)` — photos **queued for moving**, not actually moved. The destination directory stays empty until the move phase runs. Label the counter clearly to avoid confusion:
|
||
|
||
```python
|
||
# During classification:
|
||
print(f" queued for move: {len(to_move)}", flush=True)
|
||
|
||
# During move phase:
|
||
moved = 0
|
||
for ap in to_move:
|
||
shutil.move(ap, dest)
|
||
moved += 1
|
||
print(f" Actually moved: {moved}", flush=True)
|
||
```
|
||
|
||
### Pitfall: Gateway restart kills agent-child processes
|
||
|
||
Agent-launched processes (even nohup'd) are children of the gateway process tree. Restarting the gateway orphans/kills them. For work that must survive restarts, always use a cron job with `no_agent=True` (Pattern A).
|
||
|
||
## Video Content Detection and Renaming
|
||
|
||
When dealing with recovered or generically-named video files (e.g., FILE000.MOV, FILE002.MOV from data recovery tools), you can auto-detect their content using YOLO on extracted frames and rename them descriptively.
|
||
|
||
### Workflow
|
||
|
||
1. Extract a single frame from each video at ~3 seconds
|
||
2. Run YOLO on CPU (since video processing doesn't benefit from GPU for single-frame extraction)
|
||
3. Build a descriptive filename from detected objects
|
||
4. Handle collisions, blank frames, and edge cases
|
||
|
||
### Full Renaming Script
|
||
|
||
```python
|
||
import os, subprocess, json
|
||
from PIL import Image
|
||
from ultralytics import YOLO
|
||
|
||
DIR = "/mnt/drive/Videos/Recovered/MOV Multimedia file"
|
||
model = YOLO("yolov8n.pt")
|
||
|
||
files = sorted([f for f in os.listdir(DIR)
|
||
if f.upper().endswith(('.MOV', '.MP4', '.3GP', '.AVI'))
|
||
and f.upper().startswith("FILE")])
|
||
|
||
for i, fname in enumerate(files):
|
||
src = os.path.join(DIR, fname)
|
||
base, ext = os.path.splitext(fname)
|
||
|
||
# Extract frame at 3 seconds (fall back to 1s for short clips)
|
||
frame_path = "/tmp/_vf.jpg"
|
||
for ss in ["00:00:03", "00:00:01"]:
|
||
r = subprocess.run(["ffmpeg", "-y", "-ss", ss, "-i", src,
|
||
"-vframes", "1", "-q:v", "2", frame_path],
|
||
capture_output=True, timeout=20)
|
||
if os.path.exists(frame_path) and os.path.getsize(frame_path) > 1000:
|
||
break
|
||
|
||
# Run YOLO on extracted frame
|
||
classes = []
|
||
if os.path.exists(frame_path):
|
||
try:
|
||
img = Image.open(frame_path)
|
||
results = model(img, device="cpu", verbose=False)
|
||
classes = [model.names[int(b.cls)] for b in results[0].boxes]
|
||
os.remove(frame_path)
|
||
except:
|
||
pass
|
||
|
||
# Build descriptive name
|
||
if classes:
|
||
people = classes.count("person")
|
||
other = [c for c in sorted(set(classes)) if c != "person"]
|
||
parts = []
|
||
if people == 1:
|
||
parts.append("person")
|
||
elif people > 1:
|
||
parts.append(f"{people}_people")
|
||
parts.extend(other[:3]) # max 3 other objects
|
||
new_name = "_".join(parts).replace(" ", "_") + ext.lower()
|
||
else:
|
||
new_name = f"scene{ext.lower()}" # blank/dark/no detections
|
||
|
||
# Avoid filename collisions
|
||
dst = os.path.join(DIR, new_name)
|
||
counter = 1
|
||
while os.path.exists(dst):
|
||
name_no_ext = os.path.splitext(new_name)[0]
|
||
dst = os.path.join(DIR, f"{name_no_ext}_{counter}{ext.lower()}")
|
||
counter += 1
|
||
|
||
os.rename(src, dst)
|
||
if (i+1) % 10 == 0:
|
||
print(f" [{i+1}/{len(files)}]", flush=True)
|
||
```
|
||
|
||
### Real-World Renaming Examples (80 recovered files)
|
||
|
||
| Old Name | New Name | Detected |
|
||
|----------|----------|----------|
|
||
| FILE000.MOV | `chair_clock_tv.mov` | Indoor scene with TV |
|
||
| FILE054.MOV | `bench_car_elephant.mov` | Outdoor with animals |
|
||
| FILE125.MOV | `5_people_cake_cell_phone_dining_table.mov` | Party/birthday |
|
||
| FILE214.MOV | `person_dog.mov` | Person walking dog |
|
||
| FILE021.MOV | `2_people_chair_surfboard.mov` | Beach/outdoor |
|
||
| FILE176.MOV | `kite.mov` | Kite flying |
|
||
| FILE095.MOV | `scene.mov` | Blank/dark frame |
|
||
|
||
### Expected Results Distribution
|
||
|
||
On 80 recovered video files from a data recovery tool:
|
||
- **~60 files** get descriptive names (people, animals, objects detected)
|
||
- **~20 files** become `scene.mov` (blank/dark frames — common in recovered data where keyframes didn't survive)
|
||
- **~0-2 files** hit errors (truncated files, unreadable headers)
|
||
|
||
### Performance
|
||
|
||
- **~3-5 seconds per file** on CPU (i7-10700K): ffmpeg frame extraction (~1s) + YOLOv8n inference (~2-4s)
|
||
- **~4-7 minutes total** for 80 files
|
||
- GPU doesn't help here — ffmpeg decoding and single-frame inference are CPU-bound at this scale
|
||
|
||
### Pitfalls
|
||
|
||
- **Blank/dark frames at 3s**: Some recovered videos have corruption at specific timestamps. The fallback to 1s catches most of these. If both fail, the file becomes `scene.mov` — still identifiable as having no content.
|
||
- **`.3GP` extension**: Data recovery tools often produce .3GP files alongside .MOV. These are valid video containers. Include them in the file filter.
|
||
- **Short clips**: Videos under 3 seconds cause ffmpeg to warn about invalid timestamp but still produce a frame from the beginning. The `ss` flag gracefully clips to available duration.
|
||
- **Collision avoidance**: Multiple files with the same detected objects (e.g., 3 different `person.mov` files) get `_1`, `_2` suffixes. The inner `while` loop checks existence and increments.
|
||
- **Don't rename twice**: Files already renamed (no "FILE" prefix) will be skipped by the `startswith("FILE")` guard. Always keep this guard for idempotency.
|
||
- **Broken file headers**: Truncated MOV files may cause ffmpeg to hang. Always set `timeout=20` on subprocess calls.
|
||
|
||
## Performance Estimates (i7-10700K, 16 threads)
|
||
|
||
| Engine | img/s (1 worker) | img/s (8 workers) | 25k images |
|
||
|---|---|---|---|
|
||
| PyTorch CPU | 0.3 | — | ~23h |
|
||
| OpenVINO | 1.5 | ~8-10 | ~4.5h |
|
||
| PyTorch CUDA (if supported) | 30+ | — | ~15min |
|
||
|
||
## Reference Files
|
||
|
||
- `references/immich-yolo-classifier.md` — Immich-specific YOLO classifier query + path mapping.
|
||
- `references/immich-classification-pipeline.md` — Full Immich photo classification pipeline: DB query, EXIF-aware multi-orientation inference, immediate-move pattern, content categorization, flat-folder sorting. (Absorbed from `immich-photo-classification`.)
|
||
- `references/video-content-renaming.md` — YOLO-based video frame detection and auto-renaming for recovered files.
|
||
- `templates/yolo_no_people_filter.py` — GPU batch scan template: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).
|