Files

T

2026-07-12 10:17:17 -04:00

37 KiB

Raw Permalink Blame History

name, description

name	description
computer-vision-inference	Run YOLO and other CV models on self-hosted servers for photo/video classification. Covers GPU compatibility checks, OpenVINO fallback, batch processing patterns, and integration with media servers (Immich).

Computer Vision Inference on Self-Hosted Infrastructure

User Preference: Plan Before Executing

When the user asks you to do a task, stop and think before running any command. Don't default to the first approach that comes to mind. Evaluate the most efficient way: consider the bottleneck (disk I/O, GPU, CPU), the right tools (dcraw, OpenVINO, CUDA), and whether the plan could fail. Getting the approach right upfront saves hours of rework. This applies to every task, not just CV — it's a general approach preference.

Run YOLO object detection on large photo collections (15k–50k+ images) on a homelab server.

GPU Compatibility Check (First Step)

Before installing anything, verify the GPU can actually run modern ML frameworks:

# Check GPU and compute capability (CC)
nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader
# OR
nvidia-smi --query-gpu=name --format=csv,noheader
# Check CC at: https://developer.nvidia.com/cuda-gpus

CC compatibility table:

Architecture	CC Range	PyTorch CUDA 12.x	CUDA 13.0+	Notes
Turing (RTX 20xx)	7.5	✅ Supported	✅ Supported	Minimum for modern PyTorch
Ampere (RTX 30xx)	8.x	✅ Supported	✅ Supported
Ada Lovelace (RTX 40xx)	8.9	✅ Supported	✅ Supported
Pascal (GTX 10xx)	6.x	✅ Works (CUDA 12.4/12.6)	❌ Dropped	sm_61 removed from CUDA 13.0 kernels

⚠️ Python 3.14 trap (Ubuntu 26.04): The system Python 3.14 ships only PyTorch 2.12.0 wheels, which are compiled for CUDA 13.0. Pascal GPUs get no kernel image is available errors. Fix: install Python 3.12 from deadsnakes PPA and use CUDA 12.4 PyTorch wheels.

YOLO Model Sizing by VRAM

Model	Params	VRAM (batch=2, 640px)	Min GPU
YOLOv8n	3.2M	~400 MB	2 GB
YOLOv8s	11.2M	~600 MB	2 GB
YOLOv8m	25.9M	~1.2 GB	3 GB
YOLOv8l	43.7M	~2 GB	4 GB
YOLOv8x	68.2M	~3.5 GB	6 GB

4GB card (GTX 1050 Ti / 1060): Use YOLOv8s (safe) or YOLOv8m (tight but works). YOLOv8x will OOM at batch 2+. YOLOv8n is fastest but less accurate on small objects.

Installation

Option A: OpenVINO (for Intel CPUs or Pascal+ GPUs)

# Create venv on a partition with space (not /tmp)
sudo mkdir -p /mnt/storage/yolo_venv
sudo chown -R $USER:$USER /mnt/storage/yolo_venv
python3 -m venv /mnt/storage/yolo_venv
/mnt/storage/yolo_venv/bin/pip install ultralytics

# Export YOLO model to OpenVINO format
/mnt/storage/yolo_venv/bin/python3 -c "
from ultralytics import YOLO
model = YOLO('yolov8n.pt')
model.export(format='openvino', device='cpu')
"
# Saved as ~/yolov8n_openvino_model/

Option B: PyTorch CUDA (for CC ≥ 7.5 GPUs)

python3 -m venv /mnt/storage/yolo_venv
/mnt/storage/yolo_venv/bin/pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
/mnt/storage/yolo_venv/bin/pip install ultralytics

Option C: PyTorch CUDA for Pascal GPUs (CC 6.x -- GTX 1050 Ti, GTX 1060, etc.)

Pascal GPUs (CC 6.x) have no official PyTorch CUDA support in CUDA 13.0+. On Ubuntu 26.04 the system Python 3.14 ships ONLY PyTorch 2.12.0 with CUDA 13.0 — Pascal will fail with no kernel image is available for execution on the device.

Approach 1: Python 3.12 + CUDA 12.4 (recommended for Ubuntu 26.04)

Install Python 3.12 from deadsnakes PPA, create a venv, and use CUDA 12.4 PyTorch wheels:

# Install Python 3.12 (Ubuntu 26.04)
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt update && sudo apt install -y python3.12 python3.12-venv

# Create venv and install PyTorch CUDA 12.4
python3.12 -m venv /mnt/storage/yolo_venv_cu124
source /mnt/storage/yolo_venv_cu124/bin/activate
pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 \
  --index-url https://download.pytorch.org/whl/cu124
pip install ultralytics pillow numpy

# Verify
python3 -c "import torch; print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')"

Approach 2: Python 3.10 + PyTorch 2.0.1 + CUDA 11.8 (legacy stable)

# 1. Install uv (Python 3.14+ system with PEP 668? uv bypasses it)
pip3 install uv 2>&1 || /mnt/storage/yolo_venv/bin/pip install uv

# 2. Install Python 3.10
export PATH="$HOME/.local/bin:$PATH"
uv python install 3.10

# 3. Create venv with Python 3.10
uv venv --python 3.10 /path/to/gpu_venv

# 4. Install PyTorch 2.0.1+cu118 (supports CUDA 11.8 + CC 6.1)
uv pip install --python /path/to/gpu_venv/bin/python \
  torch==2.0.1+cu118 torchvision==0.15.2+cu118 \
  --index-url https://download.pytorch.org/whl/cu118

# 5. Pin numpy < 2 (PyTorch 2.0.1 was compiled against numpy 1.x)
uv pip install --python /path/to/gpu_venv/bin/python "numpy<2"

# 6. Install ultralytics
uv pip install --python /path/to/gpu_venv/bin/python ultralytics
# If pi-heif fails via uv, install pip first then use pip:
uv pip install --python /path/to/gpu_venv/bin/python pip
/path/to/gpu_venv/bin/pip install pi-heif

# 7. Verify GPU
/path/to/gpu_venv/bin/python3 -c "
import torch
print(f'CUDA: {torch.cuda.is_available()}, GPU: {torch.cuda.get_device_name(0)}')
t = torch.zeros(1,3,640,640).cuda()
conv = torch.nn.Conv2d(3, 16, 3).cuda()
print(f'CUDA compute: OK ({conv(t).shape})')
"

Performance reference (GTX 1050 Ti, USB HDD): ~4–6 img/s at batch=4. Bottleneck is disk I/O, not GPU.

YOLO Inference Patterns

Templates available:

templates/yolo_no_people_filter.py — GPU batch scan: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).

Single image (test):

from ultralytics import YOLO
model = YOLO("yolov8n_openvino_model/")
results = model("/path/to/photo.jpg", verbose=False)
for r in results:
    if r.boxes:
        for box in r.boxes:
            cls = int(box.cls[0])
            name = r.names[cls]
            conf = float(box.conf[0])
            print(f"  {name}: {conf:.2f}")

EXIF-aware inference (fixes sideways photos with orientation flags)

YOLO loads images via OpenCV (cv2.imread) which ignores EXIF orientation flags. A portrait photo taken with a phone appears rotated 90° to the model, causing missed person/object detections. Fix by pre-rotating with PIL. However, this only catches photos that HAVE EXIF orientation flags — some photos are stored with rotated pixel data and no orientation flag at all. See the section below for the brute-force fallback.

from PIL import Image, ImageOps
import numpy as np

def load_image_exif_aware(path: str) -> np.ndarray:
    """Load image with EXIF rotation applied, return BGR numpy array.
    
    Note: ImageOps.exif_transpose returns None if the image has 
    no EXIF orientation tag. Always handle None by reloading fresh.
    """
    with Image.open(path) as img:
        img = ImageOps.exif_transpose(img)
        if img is None:
            # No EXIF orientation data — reload fresh 
            return None  # Caller should fall back to brute-force rotation
        if img.mode == "RGBA":
            img = img.convert("RGB")
        elif img.mode != "RGB":
            img = img.convert("RGB")
        return np.array(img)[:, :, ::-1]  # RGB -> BGR for YOLO

img_np = load_image_exif_aware("/path/to/photo.jpg")
if img_np is not None:
    results = model(img_np, verbose=False)

Performance: PIL load + EXIF transpose is fast (~11 img/s on GTX 1050 Ti) — the rotate is a metadata operation, not a pixel resample.

Multi-Orientation Detection (Critical for Rotated Photos)

YOLO's default pipeline (OpenCV cv2.imread) ignores EXIF orientation flags and stores pixels as-is. A portrait phone photo appears 90° rotated to the model, causing missed detections. Fixing orientation uncovered 280+ missed people photos in a 4,000-photo corpus.

Two-Layer Strategy

Layer 1: EXIF-aware PIL loading (fast, catches photos with orientation flags) Layer 2: 4-orientation brute force (definitive, catches photos stored rotated without EXIF flags)

Layer	Speed	Found on 4k photos	Time
EXIF-only (PIL + `ImageOps.exif_transpose`)	~11 img/s	0 missed photos	~6 min
4-orientation brute force	~4.5 img/s	280+ missed photos	~25 min

Key takeaway: EXIF-only found nothing. 4-orientation found massive numbers of missed people photos. Many phones and apps (especially older Android, TikTok screenshots, and re-saved images) store rotated pixel data without setting the EXIF orientation flag. EXIF handling alone is insufficient.

Helper: try_all_orientations()

The definitive approach: rotate the image array 4 ways and check each for people (class 0).

import numpy as np
from PIL import Image

def try_all_orientations(ap: str, model) -> tuple[bool, str]:
    """Check all 4 rotations of a photo for people. Returns (found_person, orientation_name)."""
    try:
        with Image.open(ap) as img:
            if img.mode != 'RGB':
                img = img.convert('RGB')
            img_np = np.array(img)[:, :, ::-1]  # RGB -> BGR for YOLO
    except Exception:
        return (False, "")

    orientations = {
        '0°': img_np,
        '90°': np.rot90(img_np, k=3),    # 90° CW
        '180°': np.rot90(img_np, k=2),   # 180°
        '270°': np.rot90(img_np, k=1),   # 270° CW
    }

    for orient_name, orient_img in orientations.items():
        try:
            results = model(orient_img, device="cuda:0", verbose=False)
        except:
            continue
        for r in results:
            if r.boxes:
                for box in r.boxes:
                    if int(box.cls[0]) == 0:  # person
                        return (True, orient_name)
    return (False, "")

# Usage in restore loop:
found, orientation = try_all_orientations(ap, model)
if found:
    print(f"Found person at {orientation}")
    shutil.move(ap, original_path)

Performance: ~4.5 img/s (4× slower than single-orientation). ~25 min for 4,000 photos. RAM stays stable (tested on GTX 1050 Ti).

Full dual-pass workflow

# Pass 1: EXIF-aware (fast pre-filter)
for ap in moved_photos:
    img_bgr = load_with_exif(ap)       # tries EXIF transpose
    if img_bgr is None:
        continue                        # nothing to see RAW — pass to brute force
    results = model(img_bgr, verbose=False)
    if has_person(results):
        restore(ap)

# Pass 2: 4-orientation (catches everything else)
for ap in still_in_moved_photos:
    found, orient = try_all_orientations(ap, model)
    if found:
        restore(ap)

Practical note: Do pass 2 even if pass 1 found nothing — on a real corpus, pass 1 found 0 but pass 2 found 280+. The EXIF orientation flag is simply not present in many stored photos.

Edge cases when loading images via PIL

Problem	Cause	Fix
RGBA PNG	PNG screenshots with alpha channel produce 4-channel arrays	`img = img.convert("RGB")`
CMYK/grayscale	Some scanned images or legacy formats	`img = img.convert("RGB")`
Decompression bomb (200MP+ panoramas)	Pillow error: "Image size exceeds limit"	`PIL.Image.MAX_IMAGE_PIXELS = None` or catch `PIL.DecompressionBombError` and skip; the latter is safer
HEIC/HEIF	iPhone photos	`pip install pi-heif`
Nikon RAW (NEF)	DSLR RAW files from Nikon cameras — 20-24MB each, 6000x4000px	Install `dcraw` (`sudo apt install dcraw`) for command-line conversion. Use `dcraw -c -w <file.NEF>` to pipe decoded PPM to stdout. Do NOT use rawpy/LibRaw — it fails on many camera-specific NEF variants (especially Nikon D5200+ and other consumer bodies). dcraw is the gold standard. Example: `dcraw -c -w photo.NEF
Truncated JPEG	Corrupt or partial downloads	Wrap in try/except, skip on error
Missing EXIF orientation flag	Photo pixels are rotated but no EXIF flag set	Use 4-orientation brute force detection (see multi-orientation section)

Immediate-move pattern (files moved as classified)

When the user wants photos moved/purged as they're processed (not batched at the end), move inside the classification loop:

moved_total = 0
for i, ap in enumerate(paths):
    results = model(ap, device="cuda:0", verbose=False)

    should_move = False
    # ... classify results ...

    if should_move:
        try:
            dest = ap.replace(PHOTO_DIR, NO_PEOPLE_DIR)
            os.makedirs(os.path.dirname(dest), exist_ok=True)
            shutil.move(ap, dest)
            moved_total += 1
        except Exception as e:
            log(f"MOVE FAILED: {ap} -> {e}")

Important: Unlike the batched pattern where "moved: X" means "queued for move", in immediate mode moved_total is the actual count of files already in the destination folder.

Multiprocessing batch (use `if name == "main"` guard):

import multiprocessing

def worker(args):
    model_path, paths = args
    from ultralytics import YOLO
    import warnings
    warnings.filterwarnings("ignore")
    model = YOLO(model_path, task="detect")
    results = []
    for ap in paths:
        r = model(ap, verbose=False)
        results.append((ap, decision))
    return results

if __name__ == "__main__":
    multiprocessing.set_start_method("forkserver", force=True)
    num_workers = min(multiprocessing.cpu_count(), 8)
    chunks = [all_paths[i::num_workers] for i in range(num_workers)]
    chunk_args = [(model_path, c) for c in chunks]
    with multiprocessing.Pool(num_workers) as pool:
        all_results = pool.map(worker, chunk_args)

COCO Class Classification Heuristic

PERSON_CLASS = 0
NATURE_CLASSES = {16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 58, 77, 80}
MOVE_CLASSES = {56, 57, 59, 60, 61, 62, 63, 64, 65, 67, 70, 71, 72, 73, 74, 75, 76, 78, 79,
                1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}

Rules: Person→keep, Nature→keep, Indoor/urban→move, No detections→check file size (<400KB→move)

Integration with Immich

Query asset paths from PostgreSQL:

DB_QUERY = '''docker exec immich_postgres psql -U postgres -d immich -t -A -c "SELECT \\\"originalPath\\\" FROM asset WHERE type='IMAGE' AND visibility='timeline' AND \\\"originalPath\\\" LIKE '/data/library/%' ORDER BY \\\"originalPath\\\";" 2>/dev/null'''

Map virtual paths to filesystem:

PHOTO_BASE = "/mnt/wd-passport/immich/photos"
def map_path(db_path):
    if db_path.startswith("/data/"):
        return PHOTO_BASE + "/" + db_path[6:]
    elif db_path.startswith("/external/"):
        return "/mnt/wd-passport" + db_path[8:]
    return db_path

Flatten sorted photos into a single folder

After classification, photos end up scattered across Immich's subdirectory tree (e.g., NO PEOPLE PHOTOS/library/4defcb72-.../2026/2026-04-07/photo.jpg). To flatten them into a single folder with no subdirectories:

import os, shutil

NO_PEOPLE = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS"
TARGET = os.path.join(NO_PEOPLE, "ALL Photos")
os.makedirs(TARGET, exist_ok=True)

# Find all files excluding the target itself
files = []
for root, dirs, fnames in os.walk(NO_PEOPLE):
    if root.startswith(TARGET):
        continue
    for f in fnames:
        files.append(os.path.join(root, f))

moved = 0
collisions = 0
for src in files:
    fname = os.path.basename(src)
    dst = os.path.join(TARGET, fname)
    if os.path.exists(dst):
        base, ext = os.path.splitext(fname)
        n = 1
        while os.path.exists(os.path.join(TARGET, f"{base}_{n}{ext}")):
            n += 1
        dst = os.path.join(TARGET, f"{base}_{n}{ext}")
        collisions += 1
    shutil.move(src, dst)
    moved += 1

print(f"Moved {moved} files, {collisions} renamed for collision")

# Clean up empty directories
for root, dirs, fnames in os.walk(NO_PEOPLE, topdown=False):
    if root != TARGET and not os.listdir(root) and root != NO_PEOPLE:
        os.rmdir(root)

Expectation: 3,600+ files flatten in seconds. Filename collisions are rare (~1 in 3,600). Collision handling appends _1, _2, etc. before the extension.

Categorization by Detected Content

After sorting photos out (no people, no scenery), you may want to further categorize the remaining photos by what objects they actually contain. Run YOLO on every photo and build a detection histogram, then let the user decide which categories are worthwhile and which are trash.

Detection Histogram Script

import os, numpy as np
from PIL import Image
from collections import Counter
from ultralytics import YOLO

ALL_PHOTOS = "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
model = YOLO("yolov8n.pt")

# Gather all image files
files = sorted([os.path.join(ALL_PHOTOS, f) for f in os.listdir(ALL_PHOTOS)
                if f.lower().endswith(('.jpg', '.jpeg', '.png', '.heic', '.heif'))])

detection_counts = Counter()  # class_id -> count of photos containing it
no_detect = 0

for i, ap in enumerate(files):
    if i > 0 and i % 300 == 0:
        print(f"  [{i}/{len(files)}] ... no_detect: {no_detect}", flush=True)

    # Load with PIL (handles EXIF + RGBA)
    with Image.open(ap) as img:
        if img.mode != 'RGB':
            img = img.convert('RGB')
        img_np = np.array(img)[:, :, ::-1]

    results = model(img_np, device="cuda:0", verbose=False)

    classes_found = set()
    for r in results:
        if r.boxes:
            for box in r.boxes:
                classes_found.add(int(box.cls[0]))

    if classes_found:
        for c in classes_found:
            detection_counts[c] += 1
    else:
        no_detect += 1

# Print top results
CLASS_NAMES = {0:"person",1:"bicycle",2:"car",3:"motorcycle",15:"cat",16:"dog",
               39:"bottle",41:"cup",56:"chair",62:"tv",63:"laptop",67:"cell phone",
               73:"book",74:"clock"}  # add more as needed
for cls_id, count in detection_counts.most_common(20):
    name = CLASS_NAMES.get(cls_id, f"c{cls_id}")
    print(f"  {name:20s}: {count:4d} photos ({count/len(files)*100:.1f}%)")
print(f"  {'no detection':20s}: {no_detect:4d} photos ({no_detect/len(files)*100:.1f}%)")

Real-World Detection Distribution (3,668 photos sorted out as "no people/scenery")

Object	Photos	%	Likely
book	668	18%	Documents, receipts, textbooks
cell phone	342	9%	Phone screenshots
tv/monitor	320	9%	Screen captures
laptop	274	7%	Computer screenshots
car	265	7%	Vehicle photos
refrigerator	218	6%	Fridge contents
clock	128	3%	Timestamps
chair	102	3%	Random furniture
dining table	74	2%	Meals
toilet	74	2%	Bathroom shots
airplane	54	1.5%	Travel
No detection	899	25%	Blank/blurry/text-only

Sorting Into Subfolders by Category

After the histogram gives you a picture of what's there, present it to the user and ask which categories to keep vs discard. Then sort:

import os, shutil

TRASH_CLASSES = {62, 63, 67, 73}  # tv, laptop, cell phone, book
TRASH_DIR = os.path.join(ALL_PHOTOS, "..", "Trash Screenshots")
os.makedirs(TRASH_DIR, exist_ok=True)

KEEP_CLASSES = {2, 5, 4, 39, 41, 74}  # car, bus, airplane, bottle, cup, clock
KEEP_DIR = os.path.join(ALL_PHOTOS, "..", "Maybe Keep")
os.makedirs(KEEP_DIR, exist_ok=True)

for fname in os.listdir(ALL_PHOTOS):
    ap = os.path.join(ALL_PHOTOS, fname)
    if not os.path.isfile(ap):
        continue
    # ... check fname against a pre-computed CSV of classes ...
    shutil.move(ap, dest)

Practical tip: Instead of re-running YOLO on every photo, save the classification results to a CSV during the detection histogram pass. Then use the CSV to sort — much faster:

# During detection pass, write:
#   all_photo_categories.csv
#   filename,detected_classes
with open("all_photo_categories.csv", "w") as f:
    f.write("filename,detected_classes\n")
    for fname, classes in photo_classes.items():
        names = ";".join(sorted(CLASS_NAMES.get(c, f"c{c}") for c in classes))
        f.write(f"{fname},{names}\n")
    # Also record no-detection photos
    for fname in no_detect_files:
        f.write(f"{fname},none\n")

Then sort using the CSV — instant classification lookup, no GPU needed.

Pitfalls

HEIC photos: pip install pi-heif for HEIC/HEIF support
Slow USB drives: Use PostgreSQL DB query instead of filesystem find/walk on external drives with 20k+ files
Docker bridge IP loss: If bridge lacks gateway IP, do full docker compose down && docker compose up -d
OpenVINO batching: LATENCY mode won't batch — use multiprocessing for parallelism
Venv placement: PyTorch+ultralytics is ~2-3GB. Install on a data partition, not /tmp
Multiprocessing: Must use if __name__ == "__main__" guard with set_start_method("forkserver") in Python 3.14+
stream=True to prevent OOM: Without stream=True, YOLO holds all results in RAM across the entire run. For 15k+ images, each result with boxes metadata accumulates until the process is killed by OOM killer. Always pass stream=True to model(ap, verbose=False, stream=True) when iterating through many images. Note: stream=True returns a generator; materialize it with list() or iterate immediately.
Zombie GPU processes: Killing a YOLO script (Ctrl+C, kill, or OOM crash) often leaves a zombie Python process holding all VRAM. The GPU shows 4024MiB / 4096MiB used but no visible process. Run nvidia-smi to find the PID, then kill -9 <pid>. Always check nvidia-smi after a failed run — the GPU won't be usable until the zombie is cleared.
Cascading CUDA failures after zombie: If a new YOLO run immediately OOMs on a model that previously fit (e.g. YOLOv8s on 4GB), check nvidia-smi for a ghost process from a prior run. The "out of memory" message is correct — the GPU IS full, just by a process that appears dead.
Corrupt file batch fallback trap: When a YOLO batch fails due to corrupt images, the naive fallback re-tries each image individually on GPU at reduced resolution. On a 4GB card with dozens of corrupt files (common in recovery data), this can stall the pipeline for hours — each retry re-loads the model, allocates CUDA context, and fails. Fix: treat batch-level failures as a corrupt batch. Move all images in the failed batch directly to the destination without any GPU retry. If the user asks to classify corrupt images separately, use PIL to check for readability first (fast CPU operation), then only run YOLO on readable files.
Background output buffering via SSH: When launching a long-running Python script via terminal(background=true) over SSH, standard output is often completely invisible — the SSH pipe buffers aggressively and the process viewer shows empty output for minutes. Fix: redirect stdout+stderr to a log file on the remote server (> /tmp/script.log 2>&1), then monitor the log file with read_file(). Also use python3 -u (unbuffered mode) and flush=True on all print() calls.
NumPy version mismatch: PyTorch 2.0.1 was compiled against numpy 1.x. Installing ultralytics pulls in numpy 2.x, causing RuntimeError: Numpy is not available at inference time. Pin numpy<2 in the venv after installing everything else.
setsid for background isolation: nohup is insufficient for SSH-launched background processes — setsid detaches from the terminal session entirely, preventing SIGHUP from the SSH closure. Use: setsid python /tmp/script.py > /home/ray/log 2>&1 &

GPU Memory FAQ

Users frequently ask "why is my GPU only using X MB when it has Y GB available?" during YOLO inference:

Why VRAM usage stays low (e.g. 432MB / 4GB)

Factor	Why
Model size	YOLOv8n (nano) is ~6.3MB. Weights + CUDA overhead = ~200–500MB VRAM. YOLOv8x would use ~1–2GB
Batch size = 1	Each frame is processed and discarded before the next. No gradient history, no activation cache (those are training-only)
Inference ≠ Training	Training batches many images, backpropagates through the full graph, and caches activations for gradient computation — that's what fills VRAM. Inference is forward-pass-only
CUDA context floor	PyTorch reserves ~300–500MB of VRAM just for the CUDA runtime context regardless of model size

The 1050 Ti's 4GB is sitting idle because the workload doesn't need more — not because something's misconfigured. YOLOv8n at batch-1 on a single image simply doesn't fill a 4GB buffer.

Why system RAM is high even with GPU inference

If a YOLO process shows 3.7GB RSS in ps aux but only 432MB in nvidia-smi:

Layer	RAM
Python + PyTorch + Ultralytics + CUDA libs loaded in memory	~1.5 GB
Image decode buffers (reads JPEGs from disk into pixel arrays)	varies
PyTorch pre-allocated CPU tensor arena	~500 MB
CUDA driver / runtime bookkeeping on CPU side	~200 MB
Python overhead (file path list, results dicts for 15k+ images)	~200 MB

The GPU is the calculator, but the CPU is the assistant that reads files from disk, decodes JPEGs, normalizes pixels, and copies tensors over. The assistant needs a desk (RAM) to work at.

Sequential Batch Processing (for disk-bound pipelines)

Comparing NEF (RAW) vs JPG Pairs with YOLO

Useful when you need to verify that paired RAW+JPEG files (same base filename in a Nikon camera's dual-save setup) are actually the same photo. The JPEG is the camera's processed output, the NEF is the sensor data — they should be the same image with different processing.

Pitfall: rawpy/LibRaw fails on many consumer NEFs

rawpy (Python binding for LibRaw) produces "data corrupted" errors on many Nikon D5200 and similar consumer-body NEF files. These errors are false positives — the files open fine in Lightroom, Nikon's software, and dcraw. Always fall back to dcraw for NEF conversion.

dcraw + YOLO comparison workflow

import subprocess, io
from PIL import Image

# Convert NEF to PPM via dcraw (piped to stdout)
r = subprocess.run(['dcraw', '-c', '-w', str(nef_path)],
                   capture_output=True, timeout=60)
if r.returncode != 0:
    raise Exception(f"dcraw failed: {r.stderr.decode()[:80]}")

# Read PPM from stdout
nef_img = Image.open(io.BytesIO(r.stdout)).convert('RGB')
jpg_img = Image.open(jpg_path).convert('RGB')

# Resize both to same dimensions for comparison
target = (640, 480)
nef_r = nef_img.resize(target, Image.LANCZOS)
jpg_r = jpg_img.resize(target, Image.LANCZOS)

# Run YOLO on CPU
from ultralytics import YOLO
model = YOLO('yolov8n.pt')
jr = model(jpg_r, device='cpu', verbose=False)
nr = model(nef_r, device='cpu', verbose=False)

# Compare detected classes
jc = sorted([model.names[int(b.cls)] for b in jr[0].boxes])
nc = sorted([model.names[int(b.cls)] for b in nr[0].boxes])

if jc == nc:
    print("Same photo — identical YOLO detections")
else:
    print(f"Different detections: JPG={jc} vs NEF={nc}")

Expected results

The NEF version typically detects more objects than the JPEG version because it has the full sensor dynamic range (24MP, deeper shadows/highlights). A JPG that sees 4 people might detect 8 in the NEF, all in shadow areas the JPG crushed. This confirms they're the same photo with richer data in the RAW version — not different photos.

NEF size reference

Camera	NEF size	JPEG size	Resolution
Nikon D5200	~23 MB	~2-8 MB	6036 × 4020

When to use CPU vs GPU for verification

If the installed PyTorch version doesn't support the GPU's compute capability (common with GTX 1050 Ti / CC 6.1 on modern PyTorch CUDA 12+), force CPU:

model(image, device='cpu', verbose=False)

CPU inference on 640×480 images with YOLOv8n is ~1-2 img/s per core. For 100 images, expect ~2-3 minutes — acceptable for verification tasks.

When the bottleneck is disk I/O (reading from a USB HDD), not GPU compute, multiprocessing doesn't help much. A simple sequential loop is simpler and more predictable:

from ultralytics import YOLO
model = YOLO("yolov8n.pt")

counts = defaultdict(int)
to_move = []
t_start = time.time()

for i, ap in enumerate(all_paths):
    if i > 0 and i % 500 == 0:
        elapsed = time.time() - t_start
        rate = i / elapsed
        eta = (len(all_paths) - i) / rate / 60
        print(f"  [{i}/{len(all_paths)}] {rate:.1f} img/s, ETA {eta:.0f}min", flush=True)

    try:
        results = model(ap, device="cuda:0", verbose=False)
    except:
        continue

    # classify results...

Performance reference (GTX 1050 Ti, USB HDD): ~6–7 img/s on 16k images, ~40 min total for 16k photos.

Running Long Inference Without Blocking the Hermes Gateway

Critical: If the agent launches a long classification script inside its conversation turn, the Telegram/Discord bot appears "typing..." and cannot process new messages until the turn finishes. Each agent iteration also makes an LLM API call — polling progress wastes tokens while the user is charged per-call.

❌ Anti-pattern: Agent launches script and waits

User: "classify my photos"
Agent: runs terminal("python classify.py")   ← blocks for 40 min
User: "what's the status?"                    ← queued, not seen
       ...bot shows typing for 40 min...
       ← agent finally responds (dozens of API calls eaten)

✅ Pattern A: Durable cron job (recommended)

Create a cron job with no_agent=True — the script runs independently, stdout goes directly to Telegram. Survives gateway restart, zero token cost during execution:

cronjob(
    action="create",
    schedule="now",
    script="/home/ray/yolo_classify.py",
    no_agent=True,
    deliver="origin"
)

✅ Pattern B: terminal(background=true) + log file + end turn

Launch with background=true, redirect stdout to a log file (SSH pipe buffers aggressively — without this, output is invisible), and immediately end the turn. Bot becomes responsive but does not survive gateway restart — child processes are killed when the gateway restarts:

# Redirect to a log file to avoid SSH pipe buffering
terminal(command="bash -c 'source venv/bin/activate && python3 -u classify.py > /tmp/classify.log 2>&1'", background=true, notify_on_complete=true)
# Monitor progress via read_file("/tmp/classify.log")

Note: Always use python3 -u (unbuffered) and flush=True on print() calls, even with log redirection. Without -u, Python's own output buffering delays log writes and read_file() shows stale data.

This user's preference: Always run long tasks in the background. They want to stay able to message me while work is processing — no waiting for 10-40 min tasks to finish before I respond.


### Pitfall: "moved: X" is queued, not actually moved

In a 2-phase script (classify all → then move all), the log shows `moved: 2393` during classification. This is `len(to_move)` — photos **queued for moving**, not actually moved. The destination directory stays empty until the move phase runs. Label the counter clearly to avoid confusion:

```python
# During classification:
print(f"  queued for move: {len(to_move)}", flush=True)

# During move phase:
moved = 0
for ap in to_move:
    shutil.move(ap, dest)
    moved += 1
print(f"  Actually moved: {moved}", flush=True)

Pitfall: Gateway restart kills agent-child processes

Agent-launched processes (even nohup'd) are children of the gateway process tree. Restarting the gateway orphans/kills them. For work that must survive restarts, always use a cron job with no_agent=True (Pattern A).

Video Content Detection and Renaming

When dealing with recovered or generically-named video files (e.g., FILE000.MOV, FILE002.MOV from data recovery tools), you can auto-detect their content using YOLO on extracted frames and rename them descriptively.

Workflow

Extract a single frame from each video at ~3 seconds
Run YOLO on CPU (since video processing doesn't benefit from GPU for single-frame extraction)
Build a descriptive filename from detected objects
Handle collisions, blank frames, and edge cases

Full Renaming Script

import os, subprocess, json
from PIL import Image
from ultralytics import YOLO

DIR = "/mnt/drive/Videos/Recovered/MOV Multimedia file"
model = YOLO("yolov8n.pt")

files = sorted([f for f in os.listdir(DIR) 
                if f.upper().endswith(('.MOV', '.MP4', '.3GP', '.AVI'))
                and f.upper().startswith("FILE")])

for i, fname in enumerate(files):
    src = os.path.join(DIR, fname)
    base, ext = os.path.splitext(fname)

    # Extract frame at 3 seconds (fall back to 1s for short clips)
    frame_path = "/tmp/_vf.jpg"
    for ss in ["00:00:03", "00:00:01"]:
        r = subprocess.run(["ffmpeg", "-y", "-ss", ss, "-i", src, 
                           "-vframes", "1", "-q:v", "2", frame_path],
                          capture_output=True, timeout=20)
        if os.path.exists(frame_path) and os.path.getsize(frame_path) > 1000:
            break

    # Run YOLO on extracted frame
    classes = []
    if os.path.exists(frame_path):
        try:
            img = Image.open(frame_path)
            results = model(img, device="cpu", verbose=False)
            classes = [model.names[int(b.cls)] for b in results[0].boxes]
            os.remove(frame_path)
        except:
            pass

    # Build descriptive name
    if classes:
        people = classes.count("person")
        other = [c for c in sorted(set(classes)) if c != "person"]
        parts = []
        if people == 1:
            parts.append("person")
        elif people > 1:
            parts.append(f"{people}_people")
        parts.extend(other[:3])  # max 3 other objects
        new_name = "_".join(parts).replace(" ", "_") + ext.lower()
    else:
        new_name = f"scene{ext.lower()}"  # blank/dark/no detections

    # Avoid filename collisions
    dst = os.path.join(DIR, new_name)
    counter = 1
    while os.path.exists(dst):
        name_no_ext = os.path.splitext(new_name)[0]
        dst = os.path.join(DIR, f"{name_no_ext}_{counter}{ext.lower()}")
        counter += 1

    os.rename(src, dst)
    if (i+1) % 10 == 0:
        print(f"  [{i+1}/{len(files)}]", flush=True)

Real-World Renaming Examples (80 recovered files)

Old Name	New Name	Detected
FILE000.MOV	`chair_clock_tv.mov`	Indoor scene with TV
FILE054.MOV	`bench_car_elephant.mov`	Outdoor with animals
FILE125.MOV	`5_people_cake_cell_phone_dining_table.mov`	Party/birthday
FILE214.MOV	`person_dog.mov`	Person walking dog
FILE021.MOV	`2_people_chair_surfboard.mov`	Beach/outdoor
FILE176.MOV	`kite.mov`	Kite flying
FILE095.MOV	`scene.mov`	Blank/dark frame

Expected Results Distribution

On 80 recovered video files from a data recovery tool:

~60 files get descriptive names (people, animals, objects detected)
~20 files become scene.mov (blank/dark frames — common in recovered data where keyframes didn't survive)
~0-2 files hit errors (truncated files, unreadable headers)

Performance

~3-5 seconds per file on CPU (i7-10700K): ffmpeg frame extraction (~1s) + YOLOv8n inference (~2-4s)
~4-7 minutes total for 80 files
GPU doesn't help here — ffmpeg decoding and single-frame inference are CPU-bound at this scale

Pitfalls

Blank/dark frames at 3s: Some recovered videos have corruption at specific timestamps. The fallback to 1s catches most of these. If both fail, the file becomes scene.mov — still identifiable as having no content.
.3GP extension: Data recovery tools often produce .3GP files alongside .MOV. These are valid video containers. Include them in the file filter.
Short clips: Videos under 3 seconds cause ffmpeg to warn about invalid timestamp but still produce a frame from the beginning. The ss flag gracefully clips to available duration.
Collision avoidance: Multiple files with the same detected objects (e.g., 3 different person.mov files) get _1, _2 suffixes. The inner while loop checks existence and increments.
Don't rename twice: Files already renamed (no "FILE" prefix) will be skipped by the startswith("FILE") guard. Always keep this guard for idempotency.
Broken file headers: Truncated MOV files may cause ffmpeg to hang. Always set timeout=20 on subprocess calls.

Performance Estimates (i7-10700K, 16 threads)

Engine	img/s (1 worker)	img/s (8 workers)	25k images
PyTorch CPU	0.3	—	~23h
OpenVINO	1.5	~8-10	~4.5h
PyTorch CUDA (if supported)	30+	—	~15min

Reference Files

references/immich-yolo-classifier.md — Immich-specific YOLO classifier query + path mapping.
references/immich-classification-pipeline.md — Full Immich photo classification pipeline: DB query, EXIF-aware multi-orientation inference, immediate-move pattern, content categorization, flat-folder sorting. (Absorbed from immich-photo-classification.)
references/video-content-renaming.md — YOLO-based video frame detection and auto-renaming for recovered files.
templates/yolo_no_people_filter.py — GPU batch scan template: keep people/scenic/screenshots, move everything else. 4GB-GPU-safe (YOLOv8s, batch=2).

37 KiB Raw Permalink Blame History Unescape Escape

Computer Vision Inference on Self-Hosted Infrastructure

User Preference: Plan Before Executing

GPU Compatibility Check (First Step)

YOLO Model Sizing by VRAM

Installation

Option A: OpenVINO (for Intel CPUs or Pascal+ GPUs)

Option B: PyTorch CUDA (for CC ≥ 7.5 GPUs)

Option C: PyTorch CUDA for Pascal GPUs (CC 6.x -- GTX 1050 Ti, GTX 1060, etc.)

YOLO Inference Patterns

Single image (test):

EXIF-aware inference (fixes sideways photos with orientation flags)

Multi-Orientation Detection (Critical for Rotated Photos)

Two-Layer Strategy

Helper: try_all_orientations()

Full dual-pass workflow

Edge cases when loading images via PIL

Immediate-move pattern (files moved as classified)

Multiprocessing batch (use if __name__ == "__main__" guard):

COCO Class Classification Heuristic

Integration with Immich

Query asset paths from PostgreSQL:

Map virtual paths to filesystem:

Flatten sorted photos into a single folder

Categorization by Detected Content

Detection Histogram Script

Real-World Detection Distribution (3,668 photos sorted out as "no people/scenery")

Sorting Into Subfolders by Category

Pitfalls

GPU Memory FAQ

Why VRAM usage stays low (e.g. 432MB / 4GB)

Why system RAM is high even with GPU inference

Sequential Batch Processing (for disk-bound pipelines)

Comparing NEF (RAW) vs JPG Pairs with YOLO

Pitfall: rawpy/LibRaw fails on many consumer NEFs

dcraw + YOLO comparison workflow

Expected results

NEF size reference

When to use CPU vs GPU for verification

Running Long Inference Without Blocking the Hermes Gateway

❌ Anti-pattern: Agent launches script and waits

✅ Pattern A: Durable cron job (recommended)

✅ Pattern B: terminal(background=true) + log file + end turn

Pitfall: Gateway restart kills agent-child processes

Video Content Detection and Renaming

Workflow

Full Renaming Script

Real-World Renaming Examples (80 recovered files)

Expected Results Distribution

Performance

Pitfalls

Performance Estimates (i7-10700K, 16 threads)

Reference Files

37 KiB

Raw Permalink Blame History

Multiprocessing batch (use `if name == "main"` guard):