initial commit
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Chrome Download Diagnostics
|
||||
|
||||
When Chrome downloads stall or get cancelled, the browser stores detailed state in a SQLite database. This is useful for diagnosing why downloads failed (e.g. Safe Browsing, network issues, etc.).
|
||||
|
||||
## Chrome's Download History DB
|
||||
|
||||
**Location:** `~/.config/google-chrome/Default/History`
|
||||
|
||||
For a Docker/Kasm container:
|
||||
```bash
|
||||
docker cp <container>:~/.config/google-chrome/Default/History /tmp/chrome_history.db
|
||||
```
|
||||
|
||||
## Key Queries
|
||||
|
||||
### All recent downloads with state + reason codes
|
||||
```python
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/tmp/chrome_history.db')
|
||||
cur = conn.cursor()
|
||||
|
||||
state_map = {
|
||||
0: 'IN_PROGRESS', 1: 'COMPLETE', 2: 'CANCELLED',
|
||||
3: 'INTERRUPTED', 4: 'INTERRUPTED_AUTO_RESUME'
|
||||
}
|
||||
reason_map = {
|
||||
0: 'No interrupt', 40: 'FILE_SECURITY_CHECK_FAILED',
|
||||
50: 'NETWORK_FAILED', 51: 'NETWORK_TIMEOUT',
|
||||
52: 'NETWORK_DISCONNECTED', 58: 'SERVER_UNAUTHORIZED',
|
||||
70: 'USER_CANCELED', 71: 'USER_SHUTDOWN', 80: 'CRASH'
|
||||
}
|
||||
danger_map = {
|
||||
0: 'SAFE', 4: 'UNCOMMON', 1: 'DANGEROUS', 2: 'DANGEROUS_URL'
|
||||
}
|
||||
|
||||
cur.execute('''
|
||||
SELECT id, target_path, state, interrupt_reason,
|
||||
total_bytes, received_bytes, danger_type, end_time
|
||||
FROM downloads ORDER BY id DESC LIMIT 15
|
||||
''')
|
||||
for r in cur.fetchall():
|
||||
sid, path, state, reason, total, recv, danger, et = r
|
||||
s = state_map.get(state, f'UNK({state})')
|
||||
name = path.split('/')[-1] if path else '(no path)'
|
||||
total_gb = f'{total//1024//1024} MB' if total else '?'
|
||||
print(f'ID {sid}: {name} → {s} ({total_gb})')
|
||||
```
|
||||
|
||||
### Check for in-progress downloads
|
||||
```sql
|
||||
SELECT * FROM downloads WHERE state=0
|
||||
```
|
||||
|
||||
## Common Error Patterns
|
||||
|
||||
| Scenario | State | Reason Code | Danger |
|
||||
|----------|-------|-------------|--------|
|
||||
| Large file killed by Safe Browsing | CANCELLED (2) | 40 (FILE_SECURITY_CHECK_FAILED) | 4 (UNCOMMON) |
|
||||
| Network dropped mid-download | INTERRUPTED (3) | 52 (NETWORK_DISCONNECTED) | 0 |
|
||||
| User cancelled | CANCELLED (2) | 70 (USER_CANCELED) | 0 |
|
||||
| Chrome crash | INTERRUPTED (3) | 80 (CRASH) | 0 |
|
||||
|
||||
## Chrome Managed Policies (Linux)
|
||||
|
||||
Policies live in `/etc/opt/chrome/policies/managed/` for system-wide or under the user's profile. Multiple JSON files are merged automatically.
|
||||
|
||||
**To disable download scanning:**
|
||||
```json
|
||||
{
|
||||
"DownloadRestrictions": 0,
|
||||
"SafeBrowsingEnabled": false,
|
||||
"SafeBrowsingProtectionForDownloadEnabled": false
|
||||
}
|
||||
```
|
||||
|
||||
After writing the policy file, restart Chrome: `pkill -f chrome` (the Kasm container will auto-restart it).
|
||||
@@ -0,0 +1,130 @@
|
||||
# Switching Immich Facial Recognition Models
|
||||
|
||||
> How to switch between `antelopev2`, `buffalo_l`, and `buffalo_s`
|
||||
> facial recognition models — and the critical gotcha that WILL bite you.
|
||||
|
||||
## When to switch
|
||||
|
||||
- **antelopev2 → buffalo_l**: Family members with similar faces are getting
|
||||
merged/confused. buffalo_l is significantly better at distinguishing
|
||||
similar faces (larger model, better embeddings).
|
||||
- **buffalo_l → buffalo_s**: Running low on VRAM (buffalo_s is ~50MB vs
|
||||
buffalo_l's ~182MB). Trade quality for memory.
|
||||
- **Reverse (experimental)**: If buffalo_l somehow performs worse for your
|
||||
specific dataset (unlikely but possible).
|
||||
|
||||
## Procedure
|
||||
|
||||
### 1. Set the model in .env
|
||||
|
||||
```bash
|
||||
# /opt/immich/.env
|
||||
MACHINE_LEARNING_FACIAL_RECOGNITION_MODEL_NAME=buffalo_l
|
||||
```
|
||||
|
||||
Valid values: `antelopev2`, `buffalo_l`, `buffalo_s`.
|
||||
|
||||
### 2. Restart the ML container
|
||||
|
||||
```bash
|
||||
cd /opt/immich
|
||||
sudo docker compose up -d immich-machine-learning
|
||||
```
|
||||
|
||||
The new model downloads automatically on first use (~182 MB for buffalo_l).
|
||||
To pre-cache and avoid the download-on-first-request delay, see
|
||||
`references/pre-caching-ml-models.md`.
|
||||
|
||||
### 3. Check the model is in place
|
||||
|
||||
```bash
|
||||
sudo docker exec immich_machine_learning find /cache/facial-recognition -name '*.onnx'
|
||||
# Should show: /cache/facial-recognition/buffalo_l/detection/model.onnx
|
||||
# /cache/facial-recognition/buffalo_l/recognition/model.onnx
|
||||
```
|
||||
|
||||
## 🚨 CRITICAL PITFALL: Model switch does NOT re-process existing faces
|
||||
|
||||
**Changing `MACHINE_LEARNING_FACIAL_RECOGNITION_MODEL_NAME` in `.env` and
|
||||
restarting the ML container only changes which model is loaded for FUTURE
|
||||
face detection.** It does NOT touch the existing face embeddings already
|
||||
stored in the database.
|
||||
|
||||
Old `antelopev2` embeddings remain and Immich will keep using them — meaning
|
||||
face confusion persists until you explicitly re-run face detection.
|
||||
|
||||
### The fix: Re-run face detection
|
||||
|
||||
**From the UI (easiest):**
|
||||
|
||||
1. Go to Immich admin → **Administration → Jobs → Face Detection**
|
||||
2. Click **All** (NOT "Missing" — "Missing" only processes photos without
|
||||
ANY embedding, and your old antelopev2 photos already have embeddings)
|
||||
3. Wait — on GPU this takes 1-2 hours for ~23K photos
|
||||
|
||||
**Via the API:**
|
||||
|
||||
```bash
|
||||
# Login
|
||||
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email":"admin@example.com","password":"your-password"}')
|
||||
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")
|
||||
|
||||
# Trigger face detection with force=true to re-process all assets
|
||||
curl -s -X PUT "http://localhost:2283/api/jobs/faceDetection" \
|
||||
-H "Authorization: Bearer $TOK" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"command":"start","force":true}'
|
||||
|
||||
# Trigger facial recognition too
|
||||
curl -s -X PUT "http://localhost:2283/api/jobs/facialRecognition" \
|
||||
-H "Authorization: Bearer $TOK" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"command":"start","force":true}'
|
||||
```
|
||||
|
||||
### What happens after re-processing
|
||||
|
||||
- Old face clusters (People) remain but get new embeddings
|
||||
- Faces that were incorrectly merged under `antelopev2` will get separate
|
||||
clusters under `buffalo_l`
|
||||
- You may need to manually clean up: remove wrong photos from a person's
|
||||
page in the UI (buffalo_l won't auto-unmerge what antelopev2 merged —
|
||||
it just won't merge new ones incorrectly)
|
||||
|
||||
### How to know it worked
|
||||
|
||||
```bash
|
||||
# ML container should show facial recognition activity
|
||||
sudo docker logs immich_machine_learning --tail 20 | head
|
||||
|
||||
# GPU should be actively processing
|
||||
nvidia-smi
|
||||
# Expect: ~1200 MiB VRAM, 10-40% utilization
|
||||
|
||||
# Server logs should show face detection activity
|
||||
sudo docker logs immich_server --tail 50 | grep -i 'detect\|face\|person'
|
||||
```
|
||||
|
||||
## Model comparison
|
||||
|
||||
| Model | Size | Quality | VRAM (loaded) | Best for |
|
||||
|-------|------|---------|---------------|----------|
|
||||
| antelopev2 | ~100 MB | Moderate | ~100 MB | Low-resource, not picky about false merges |
|
||||
| buffalo_l | ~182 MB | High | ~182 MB | **Default choice** — best accuracy, distinguishing similar faces |
|
||||
| buffalo_s | ~50 MB | Lower | ~50 MB | Extreme VRAM constraints |
|
||||
|
||||
## Internal ML URL
|
||||
|
||||
The ML container runs as `immich_machine_learning` on the Docker network.
|
||||
The default URL Immich uses is:
|
||||
|
||||
```
|
||||
http://immich-machine-learning:3003
|
||||
```
|
||||
|
||||
This is auto-discovered via Docker DNS. If the "Machine Learning URL" field
|
||||
in Immich admin settings is blank, it's using this default — which is
|
||||
correct when ML runs on the same Docker host. Only fill it in when running
|
||||
ML on a separate machine.
|
||||
@@ -0,0 +1,191 @@
|
||||
# Google Takeout → Immich Migration
|
||||
|
||||
> Companion reference for the `immich-server` skill. Covers the end-to-end workflow: requesting the export, downloading to the server, and importing with immich-go.
|
||||
|
||||
## Step 1 — Request from Google
|
||||
|
||||
1. Go to **[takeout.google.com](https://takeout.google.com)** and sign in
|
||||
2. Click **Deselect all**, then scroll down and check only **Google Photos**
|
||||
3. Optional: click **All photo albums included** to select specific albums (defaults to all)
|
||||
4. Scroll down, click **Next step**
|
||||
5. Configure:
|
||||
- **Delivery method:** Email download link
|
||||
- **Frequency:** Export once
|
||||
- **File type:** `.zip` or `.tgz` (no preference difference for immich-go)
|
||||
- **File size:** 2GB, 10GB, or 50GB — larger = fewer parts but longer to generate
|
||||
6. Click **Create export**
|
||||
7. Google sends an email to the account when ready (anywhere from 30 minutes to a few days for large libraries)
|
||||
|
||||
## Step 2 — Transfer to Server
|
||||
|
||||
⚠️ **Can't download Takeout links from a headless server.** Google Takeout download URLs are session-authenticated — they require your logged-in Google browser session. `curl`/`wget` from a server hits the sign-in page every time.
|
||||
|
||||
**Two working approaches:**
|
||||
|
||||
### Option A — Download locally + SCP to server
|
||||
|
||||
Download each Takeout part from your **phone or computer browser** to `~/Downloads/`, then transfer over local WiFi:
|
||||
|
||||
```bash
|
||||
# From your local machine:
|
||||
scp ~/Downloads/takeout-*.zip user@192.168.x.x:~/docker/hermes/workspace/google-takeout/
|
||||
```
|
||||
|
||||
### Option B — KasmVNC Chrome container (no local download needed)
|
||||
|
||||
Spin up a browser directly on the server and download the files in-place:
|
||||
|
||||
```bash
|
||||
# On the server:
|
||||
docker run -d \
|
||||
--name=chrome \
|
||||
--shm-size=2g \
|
||||
-p 6901:6901 \
|
||||
-e VNC_PW=password \
|
||||
-e LANG=en_US.UTF-8 \
|
||||
-v ~/docker/hermes/workspace/google-takeout:/home/kasm-user/Downloads \
|
||||
kasmweb/chrome:1.16.0
|
||||
```
|
||||
|
||||
1. Open `https://192.168.x.x:6901` in your local browser
|
||||
2. Login: `kasm_user` / `password` (accept self-signed cert)
|
||||
3. Sign into Google inside the containerized Chrome
|
||||
4. Open each Takeout link from your email — files save straight to the server
|
||||
5. Done? `docker rm -f chrome`
|
||||
|
||||
> 🐛 **Don't use linuxserver/chromium** — it uses Selkies WebSocket which often gives a black screen. kasmweb/chrome is the proven workhorse.
|
||||
|
||||
#### 🛑 Pitfall: Chrome Safe Browsing kills large Takeout downloads
|
||||
|
||||
Large Takeout zips (30-50 GB each) trigger Chrome's **FILE_SECURITY_CHECK_FAILED** (reason code 40, danger type "UNCOMMON" / type 4). Chrome silently cancels the download, leaving orphan `.crdownload` files on disk that never finish.
|
||||
|
||||
**If downloads stall with .crdownload files for >1 hour, follow these steps:**
|
||||
|
||||
**a) Diagnose** — check Chrome's History SQLite DB to confirm Safe Browsing is the culprit:
|
||||
|
||||
```bash
|
||||
docker cp chrome:/home/kasm-user/.config/google-chrome/Default/History /tmp/chrome_history.db
|
||||
python3 -c "
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/tmp/chrome_history.db')
|
||||
cur = conn.cursor()
|
||||
cur.execute('SELECT id, target_path, state, interrupt_reason FROM downloads ORDER BY id DESC')
|
||||
for r in cur.fetchall():
|
||||
print(f'ID {r[0]}: {r[1] if r[1] else \"(no path)\"} → state {r[2]} reason {r[3]}')
|
||||
conn.close()
|
||||
"
|
||||
```
|
||||
|
||||
**b) Fix** — disable Safe Browsing via managed policy:
|
||||
|
||||
```bash
|
||||
docker exec -u 0 chrome sh -c 'cat > /etc/opt/chrome/policies/managed/download_safety.json << '\''EOF'\''
|
||||
{
|
||||
"DownloadRestrictions": 0,
|
||||
"SafeBrowsingEnabled": false,
|
||||
"SafeBrowsingProtectionForDownloadEnabled": false
|
||||
}
|
||||
EOF
|
||||
'
|
||||
```
|
||||
|
||||
**c) Restart Chrome** so policies take effect:
|
||||
|
||||
```bash
|
||||
docker exec -u 0 chrome pkill -f chrome
|
||||
# KasmVNC auto-restarts Chrome within seconds
|
||||
```
|
||||
|
||||
**d) Clean up orphan files:**
|
||||
|
||||
```bash
|
||||
rm -f ~/docker/hermes/workspace/google-takeout/*.crdownload
|
||||
rm -f ~/docker/hermes/workspace/google-takeout/*.tmp
|
||||
```
|
||||
|
||||
After this, reconnect to KasmVNC, re-open the Takeout download links, and retry. Chrome will no longer block them.
|
||||
|
||||
### Extract on the server
|
||||
|
||||
```bash
|
||||
cd ~/docker/hermes/workspace/google-takeout
|
||||
mkdir -p extracted
|
||||
|
||||
# Install unzip if missing
|
||||
which unzip || sudo apt install -y unzip
|
||||
|
||||
# Extract each zip into the subdirectory (cleaner than extracting in-place)
|
||||
for f in takeout-*.zip; do
|
||||
echo "Extracting $f..."
|
||||
unzip -q -o "$f" -d extracted/
|
||||
done
|
||||
|
||||
# For .tgz files:
|
||||
# for f in *.tgz; do tar -xzf "$f" -C extracted/; done
|
||||
|
||||
echo "Done — $(find extracted/ -type f | wc -l) files extracted"
|
||||
du -sh extracted/
|
||||
```
|
||||
|
||||
The extracted structure under `extracted/` will be a single `Takeout/` directory containing:
|
||||
```
|
||||
extracted/Takeout/Google Photos/
|
||||
├── Photos from 2024/
|
||||
├── Photos from 2025/
|
||||
├── <album names>/
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Step 3 — Import with immich-go
|
||||
|
||||
```bash
|
||||
# Point immich-go at the extracted directory (NOT the zip files):
|
||||
/usr/local/bin/immich-go \
|
||||
--server=http://192.168.x.x:2283 \
|
||||
--api-key=YOUR_IMMICH_KEY \
|
||||
upload from-google-photos ~/docker/hermes/workspace/google-takeout/extracted/
|
||||
|
||||
# The tool:
|
||||
# - Strips out .json metadata sidecar files automatically
|
||||
# - Preserves EXIF timestamps and dates
|
||||
# - Restores album/album structure from Google Takeout format
|
||||
# - Skips duplicates (matched by content hash)
|
||||
# - Preserves people tags and archived/trashed state
|
||||
|
||||
# If the API key lacks job.create permission, add --pause-immich-jobs=FALSE:
|
||||
/usr/local/bin/immich-go \
|
||||
--server=http://192.168.x.x:2283 \
|
||||
--api-key=YOUR_IMMICH_KEY \
|
||||
--pause-immich-jobs=FALSE \
|
||||
upload from-google-photos ~/docker/hermes/workspace/google-takeout/extracted/
|
||||
```
|
||||
|
||||
**⚠️ Flag gotchas:**
|
||||
- Use `--server` and `--api-key` (double-dash long form). Single-dash `-server` is parsed as `-s` + `erver=` and fails.
|
||||
- `upload` is a parent command — you MUST specify a subcommand: `from-google-photos` (for Takeout) or `from-folder` (for raw folders).
|
||||
|
||||
|
||||
Progress updates scroll in terminal. For large imports (38K+ files, 184GB), expect 30-60 minutes depending on server load and disk speed.
|
||||
|
||||
## Step 4 — Cleanup
|
||||
|
||||
After confirming all photos imported successfully:
|
||||
|
||||
```bash
|
||||
# Option A — just remove the extracted files, keep zips
|
||||
rm -rf ~/docker/hermes/workspace/google-takeout/extracted
|
||||
|
||||
# Option B — remove everything including zips
|
||||
rm -rf ~/docker/hermes/workspace/google-takeout
|
||||
```
|
||||
|
||||
## Common Sizes
|
||||
|
||||
| Google Photos Library | Takeout Size Estimate |
|
||||
|-----------------------|-----------------------|
|
||||
| 10 GB used | ~10-12 GB |
|
||||
| 50 GB used | ~50-60 GB |
|
||||
| 200 GB used | ~200-240 GB |
|
||||
| 2 TB used | ~2-2.5 TB |
|
||||
|
||||
Downloading large archives directly to the server (Option B) avoids any need to keep a desktop computer running overnight.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Hardware Recommendations for Immich
|
||||
|
||||
> Guidance for upgrading hardware to run Immich smoothly, especially for ML-heavy workloads (face detection, smart search, OCR).
|
||||
|
||||
## The Bottleneck: Immich ML
|
||||
|
||||
Immich's background ML jobs (smart search, face detection, facial recognition, OCR, thumbnail generation) are **CPU-intensive** on low-power hardware. Without a GPU, these jobs run entirely on the CPU and can saturate available cores for hours or days after a large import.
|
||||
|
||||
### What Each Job Consumes
|
||||
|
||||
| Job | CPU Usage | GPU-Accelerated? |
|
||||
|-----|-----------|------------------|
|
||||
| `metadataExtraction` | Light | No |
|
||||
| `thumbnailGeneration` | Medium | Yes (ffmpeg GPU encode) |
|
||||
| `smartSearch` (CLIP) | **Heavy** — 5-10 sec/image on CPU | Yes (CUDA) |
|
||||
| `faceDetection` | **Heavy** | Yes (CUDA) |
|
||||
| `facialRecognition` | **Heavy** | Yes (CUDA) |
|
||||
| `OCR` | **Heavy** | Partial |
|
||||
| `videoConversion` | **Very Heavy** — 3-5 min/video on CPU | Yes (NVENC) |
|
||||
|
||||
## Hardware Tiers
|
||||
|
||||
### 🟢 Tier 1: Low-Power Mini PC (HP EliteDesk, Dell OptiPlex Micro, Lenovo Tiny)
|
||||
|
||||
**Example:** HP EliteDesk 800 G2 DM (i5-6500T, 7GB RAM, no GPU)
|
||||
|
||||
| Pro | Con |
|
||||
|-----|-----|
|
||||
| ~15W idle, ~$65/yr power | ML jobs take **days** on CPU |
|
||||
| Silent, tiny, cheap | RAM limited (often 7-16 GB, non-upgradable) |
|
||||
| Great for basic serving | No PCIe slot — **cannot add a GPU** |
|
||||
|
||||
**Best for:** Light usage (<5K photos), no ML features needed. Accept background jobs running for days.
|
||||
|
||||
### 🟡 Tier 2: SFF Office PC (Dell OptiPlex SFF, HP EliteDesk SFF)
|
||||
|
||||
**Example:** Dell OptiPlex 3420 SFF (i7-6700, 64GB RAM)
|
||||
|
||||
| Pro | Con |
|
||||
|-----|-----|
|
||||
| 64GB RAM support | Only fits **low-profile GPUs** (rare, pricier) |
|
||||
| Slightly better CPU | Still limited cooling |
|
||||
| Cheap used ($100-150) | GPU search is a pain |
|
||||
|
||||
**GPU options (low-profile, 75W, no extra power cable):**
|
||||
- GTX 1650 LP (~$120 used) — best performance for ML
|
||||
- GT 1030 (~$50) — **avoid**, no CUDA, useless for ML
|
||||
- RTX 3050 LP (~$180) — very rare in LP form
|
||||
|
||||
### 🟢 Tier 3: Mini Tower Office PC (Dell OptiPlex MT, Precision Tower)
|
||||
|
||||
**Example:** Dell OptiPlex 7050 MT / Precision Tower 3420
|
||||
|
||||
| Pro | Con |
|
||||
|-----|-----|
|
||||
| Full PCIe slot — fits any GPU | Larger case |
|
||||
| Often has SSD bay + HDD bay | Older CPUs (6th-7th gen) |
|
||||
| Cheap used ($100-200) | Usually 16GB RAM (upgradable) |
|
||||
|
||||
**Best GPU for the money — GTX 1050 Ti 4GB (~$60-80 used):**
|
||||
- 75W — no extra power cables, runs off PCIe slot
|
||||
- 768 CUDA cores — good for YOLO and Immich ML
|
||||
- Drops ML job time from **days to hours**
|
||||
- ~$35/yr extra power over a mini PC
|
||||
|
||||
**GPU upgrade paths (all 75W, no cables needed):**
|
||||
|
||||
| GPU | Used Price | ML Speed vs 1050 Ti | VRAM | Verdict |
|
||||
|-----|-----------|---------------------|------|---------|
|
||||
| **GTX 1050 Ti** 🏆 | **$60-80** | 1x (baseline) | 4GB | Best budget pick |
|
||||
| GTX 1650 | $100 | ~1.2x | 4GB | Slightly faster, pricier |
|
||||
| **RTX 3050 6GB** 🏆 | **$150** | **~3x** (Tensor Cores) | **6GB** | Best value — ~3x faster for ~2x price |
|
||||
|
||||
### 🟣 Tier 4: Modern Desktop (Dell XPS, custom build)
|
||||
|
||||
**Example:** Dell XPS 8950 (i5-12500K, 32GB DDR5, PCIe 4.0 x16)
|
||||
|
||||
| Pro | Con |
|
||||
|-----|-----|
|
||||
| **10 cores** (6P+4E) — 3x CPU perf vs i7-6700 | More expensive |
|
||||
| DDR5 RAM — much faster memory bandwidth | Larger power draw at idle (~40W) |
|
||||
| PCIe 4.0 slot — no GPU bottleneck | |
|
||||
| Good for both serving AND processing | |
|
||||
|
||||
**With a GPU, this tier finishes Immich ML queues in ~30-60 minutes vs days on a mini PC.**
|
||||
|
||||
## Power Consumption Comparison (24/7, $0.13/kWh)
|
||||
|
||||
| Setup | Idle | Under Load | Yearly Cost |
|
||||
|-------|------|-----------|-------------|
|
||||
| Mini PC (no GPU) | ~15W | ~45W | **~$65** |
|
||||
| SFF / MT + GTX 1050 Ti | ~40W | ~140W | **~$100** |
|
||||
| SFF / MT + GTX 1650 | ~40W | ~140W | **~$100** |
|
||||
| SFF / MT + RTX 3050 | ~40W | ~145W | **~$100** |
|
||||
| Modern desktop + GPU | ~50W | ~250W | **~$140** |
|
||||
|
||||
## Quick Recommendations
|
||||
|
||||
**"I want it cheap and don't mind waiting"** → HP mini, no changes
|
||||
**"I want it faster, budget $200"** → Used OptiPlex MT + GTX 1050 Ti ($100 + $60)
|
||||
**"I want it fast, budget $400"** → OptiPlex MT + RTX 3050 6GB + SSD
|
||||
**"I want it blazing"** → Modern desktop + RTX 3050/4060
|
||||
@@ -0,0 +1,233 @@
|
||||
# Immich API: Job Management & GPU Troubleshooting
|
||||
|
||||
> Session-specific commands for triggering ML jobs and diagnosing GPU acceleration on the Immich machine-learning container (v2.7.5+).
|
||||
|
||||
## Authentication
|
||||
|
||||
### Login (get bearer token)
|
||||
|
||||
```bash
|
||||
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email":"admin@example.com","password":"your-password"}')
|
||||
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")
|
||||
```
|
||||
|
||||
The token is a bearer token, used as `Authorization: Bearer *** all subsequent API calls.
|
||||
|
||||
## Job Management
|
||||
|
||||
### List all job queues
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:2283/api/jobs -H "Authorization: Bearer ***
|
||||
```
|
||||
|
||||
Returns each job name with `queueStatus` (isPaused, isActive) and `jobCounts` (active, completed, failed, delayed, waiting, paused).
|
||||
|
||||
### Trigger a specific job
|
||||
|
||||
```bash
|
||||
curl -s -X PUT "http://localhost:2283/api/jobs/smartSearch" \
|
||||
-H "Authorization: Bearer *** \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"command":"start"}'
|
||||
```
|
||||
|
||||
To force re-process already-completed assets, use `{"command":"start","force":true}`.
|
||||
|
||||
### Available job names
|
||||
|
||||
- `smartSearch` — CLIP embeddings for natural language search
|
||||
- `faceDetection` — detect faces in photos
|
||||
- `facialRecognition` — group detected faces into people
|
||||
- `metadataExtraction` — read EXIF dates, GPS, camera info
|
||||
- `thumbnailGeneration` — create preview thumbnails
|
||||
- `videoConversion` — transcode videos for streaming
|
||||
- `ocr` — read text in photos
|
||||
- `duplicateDetection` — find duplicate assets
|
||||
- `sidecar` — process sidecar files (XMP, etc.)
|
||||
- `library` — scan library for new/changed files
|
||||
- `backupDatabase` — create DB backup
|
||||
- `notifications` — process notification queue
|
||||
- `storageTemplateMigration` — move files to organized folder structure
|
||||
|
||||
## GPU Verification
|
||||
|
||||
### Check ONNX Runtime CUDA availability inside container
|
||||
|
||||
```bash
|
||||
docker exec immich_machine_learning bash -c \
|
||||
'source /opt/venv/bin/activate && \
|
||||
python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"'
|
||||
```
|
||||
|
||||
Expected output for GPU-enabled:
|
||||
```
|
||||
GPU
|
||||
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
|
||||
```
|
||||
|
||||
### Check GPU device files in container
|
||||
|
||||
```bash
|
||||
docker exec immich_machine_learning ls -la /dev | grep nvidia
|
||||
```
|
||||
|
||||
Should show: `nvidia0`, `nvidiactl`, `nvidia-uvm`, `nvidia-uvm-tools`
|
||||
|
||||
### Verify compose GPU config is attached
|
||||
|
||||
```bash
|
||||
docker inspect immich_machine_learning --format '{{json .HostConfig.DeviceRequests}}'
|
||||
```
|
||||
|
||||
Should show: `[{"Driver":"nvidia","Count":-1,...}]` for `count: all`
|
||||
|
||||
Check runtime: `docker inspect immich_machine_learning --format '{{json .HostConfig.Runtime}}'`
|
||||
|
||||
### Immich server container has Node.js, not Python
|
||||
|
||||
For inline JSON parsing inside `docker exec immich_server` commands, use Node:
|
||||
|
||||
```bash
|
||||
# Instead of python3 -c (not available), use:
|
||||
docker exec immich_server node -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$JSON_DATA"
|
||||
```
|
||||
|
||||
## Docker Networking Recovery
|
||||
|
||||
If the Immich server cannot reach its database or redis (EHOSTUNREACH errors):
|
||||
```
|
||||
Error: connect EHOSTUNREACH 172.18.0.3:5432
|
||||
Error: connect EHOSTUNREACH 172.18.0.4:6379
|
||||
```
|
||||
|
||||
**Fix:** Restart the stack in dependency order:
|
||||
```bash
|
||||
cd /opt/immich && docker compose restart database redis
|
||||
# Wait 5s for DB/redis to be ready
|
||||
docker compose restart immich-server immich-machine-learning
|
||||
```
|
||||
|
||||
## Docker Compose GPU Configuration (v2 format)
|
||||
|
||||
**Required** for the `v2-cuda` ML image to actually access the GPU:
|
||||
|
||||
```yaml
|
||||
immich-machine-learning:
|
||||
container_name: immich_machine_learning
|
||||
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
|
||||
volumes:
|
||||
- model-cache:/cache
|
||||
env_file:
|
||||
- .env
|
||||
restart: always
|
||||
healthcheck:
|
||||
disable: false
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- driver: nvidia
|
||||
count: all
|
||||
capabilities: [gpu]
|
||||
```
|
||||
|
||||
The `v2-cuda` image already has `DEVICE=cuda` and `NVIDIA_VISIBLE_DEVICES=all` in its Dockerfile ENV — no need to add those in compose. The missing piece is the `deploy.resources.reservations.devices` block.
|
||||
|
||||
**⚠️ Container has no `curl`:** The ML container has no `curl` or `ping` installed. For network tests, use Python's `urllib.request` from inside the activated venv:
|
||||
```bash
|
||||
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
|
||||
import urllib.request
|
||||
r = urllib.request.urlopen('\''https://huggingface.co'\'', timeout=10)
|
||||
print(r.status)
|
||||
"'
|
||||
```
|
||||
|
||||
## GPU Functional Verification
|
||||
|
||||
### Layered diagnostic procedure
|
||||
|
||||
When the ML container shows models failing to load, use this progression to isolate the issue:
|
||||
|
||||
**Layer 1 — GPU hardware is exposed to container:**
|
||||
```bash
|
||||
docker exec immich_machine_learning ls -la /dev | grep nvidia
|
||||
# Must show: nvidia0, nvidiactl, nvidia-uvm
|
||||
```
|
||||
|
||||
**Layer 2 — ONNX Runtime detects CUDA providers:**
|
||||
```bash
|
||||
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
|
||||
import onnxruntime
|
||||
print(\"Device:\", onnxruntime.get_device())
|
||||
print(\"Providers:\", onnxruntime.get_available_providers())
|
||||
"'
|
||||
# Expected: Device=GPU, Providers=[TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider]
|
||||
```
|
||||
|
||||
**Layer 3 — Actual CUDA inference works:**
|
||||
```bash
|
||||
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
|
||||
import onnxruntime as ort, numpy as np, onnx
|
||||
from onnx import helper, TensorProto
|
||||
|
||||
# Build a minimal model (Relu) and run it on CUDA
|
||||
X = helper.make_tensor_value_info(\"X\", TensorProto.FLOAT, [1, 3])
|
||||
Y = helper.make_tensor_value_info(\"Y\", TensorProto.FLOAT, [1, 3])
|
||||
node = helper.make_node(\"Relu\", [\"X\"], [\"Y\"])
|
||||
graph = helper.make_graph([node], \"test\", [X], [Y])
|
||||
model = helper.make_model(graph)
|
||||
|
||||
sess = ort.InferenceSession(model.SerializeToString(),
|
||||
providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"])
|
||||
result = sess.run(None, {\"X\": np.array([[-2.0, 0.0, 2.0]], dtype=np.float32)})
|
||||
print(\"CUDA inference OK:\", result[0])
|
||||
"'
|
||||
# Expected: CUDA inference OK: [[0. 0. 2.]]
|
||||
```
|
||||
|
||||
**Layer 4 — Load a real CLIP model on GPU (after download):**
|
||||
```bash
|
||||
# Download the CLIP model first (run this step if not yet cached)
|
||||
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
|
||||
from huggingface_hub import snapshot_download, os
|
||||
cache_dir = \"/cache/clip/ViT-B-32__openai\"
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
snapshot_download(\"immich-app/ViT-B-32__openai\", cache_dir=cache_dir, local_dir=cache_dir,
|
||||
ignore_patterns=[\"*.armnn\", \"*.rknn\"], max_workers=2)
|
||||
"'
|
||||
|
||||
# Then load and run inference on GPU
|
||||
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
|
||||
import onnxruntime as ort, numpy as np, time, os
|
||||
model_path = \"/cache/clip/ViT-B-32__openai/visual/model.onnx\"
|
||||
print(f\"Model: {os.path.getsize(model_path)/1024/1024:.0f} MB\")
|
||||
|
||||
so = ort.SessionOptions()
|
||||
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
|
||||
sess = ort.InferenceSession(model_path, sess_options=so,
|
||||
providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"],
|
||||
provider_options=[{\"device_id\": \"0\", \"gpu_mem_limit\": 3*1024*1024*1024}, {}])
|
||||
print(f\"Loaded in {time.time():.0f}s\")
|
||||
|
||||
# Run one inference
|
||||
data = np.random.randn(1, 3, 224, 224).astype(np.float32)
|
||||
start = time.time()
|
||||
outputs = sess.run(None, {sess.get_inputs()[0].name: data})
|
||||
print(f\"Inference: {outputs[0].shape} in {(time.time()-start)*1000:.0f}ms\")
|
||||
"'
|
||||
# Expected: ~0.8s load, ~237ms inference, output shape (1, 512)
|
||||
```
|
||||
|
||||
### Common failures at each layer
|
||||
|
||||
| Layer | Failure Symptom | Likely Cause |
|
||||
|-------|----------------|--------------|
|
||||
| 1 | No nvidia device files | GPU not configured in compose — add `deploy.resources.reservations.devices` |
|
||||
| 1 | nvidia0 present but nvidia-smi not found | Container image has drivers but no nvidia-smi binary (normal for Immich ML image) |
|
||||
| 2 | Only CPUExecutionProvider | GPU not exposed to container (wrong runtime / no device requests) |
|
||||
| 3 | ONNX session creation fails | CUDA version mismatch (container CUDA 12.2 vs host driver) or OOM |
|
||||
| 4 | Model download OK but load fails | Disk space, corrupt download, or model file mismatch. Check /cache mounts |
|
||||
| 4 | Download loop (clear→retry→fail) | Multiple models writing to same cache_dir, overwriting each other's files. Try MACHINE_LEARNING_MODEL_ARENA=true
|
||||
@@ -0,0 +1,95 @@
|
||||
# Import Progress Reference
|
||||
|
||||
> Companion for the `immich-server` skill. Documents what to expect during and after a large immich-go import.
|
||||
|
||||
## immich-go Progress Output
|
||||
|
||||
When running immich-go `from-google-photos`, output scrolls like:
|
||||
|
||||
```
|
||||
Immich read 100%, Assets found: 18595, Upload errors: 0, Uploaded 2735
|
||||
Immich read 100%, Assets found: 18595, Upload errors: 0, Uploaded 2742
|
||||
...
|
||||
```
|
||||
|
||||
Key fields:
|
||||
- **Assets found** — total photos/videos detected (excludes .json sidecars)
|
||||
- **Uploaded** — running count of files sent to Immich
|
||||
- **Upload errors** — should stay 0; if non-zero, check the server logs
|
||||
|
||||
### Final Summary Format
|
||||
|
||||
```
|
||||
Asset Tracking Report:
|
||||
=====================
|
||||
Total Assets: 18595 (183.1 GB)
|
||||
Processed: 16590 (149.9 GB)
|
||||
Discarded: 1987 (33.2 GB)
|
||||
Errors: 0 (0 B)
|
||||
Pending: 18 (35.2 MB)
|
||||
|
||||
Event Report:
|
||||
=============
|
||||
Discovery (Assets):
|
||||
discovered image : 17805 (44.8 GB)
|
||||
discovered video : 790 (138.3 GB)
|
||||
|
||||
Discovery (Non-Assets):
|
||||
discovered sidecar : 18377 (13.8 MB)
|
||||
discovered unknown file : 1046 (-)
|
||||
discovered unsupported file : 244 (208.9 MB)
|
||||
|
||||
Asset Lifecycle (PROCESSED):
|
||||
uploaded successfully : 16401 (148.7 GB)
|
||||
server asset upgraded : 101 (907.2 MB)
|
||||
metadata updated : 16502 (355.2 MB)
|
||||
|
||||
Asset Lifecycle (DISCARDED):
|
||||
discarded local duplicate : 2062 (33.5 GB)
|
||||
|
||||
Processing Events:
|
||||
associated metadata : 18577
|
||||
missing metadata : 18
|
||||
stacked : 459
|
||||
added to album : 933
|
||||
tagged : 29069
|
||||
```
|
||||
|
||||
**What the sections mean:**
|
||||
- **Processed** vs **Discarded** — processed = newly uploaded, discarded = already existed on server (deduplicated by content hash)
|
||||
- **Pending** (18) — 18 assets that didn't reach a final state. Usually harmless edge case; those files likely uploaded but the status check didn't confirm before the tool exited. Manually verify by checking Immich's asset count after processing finishes.
|
||||
- **added to album** (933) — Google Photos albums recreated in Immich
|
||||
- **tagged** (29069) — people tags from Google Photos applied
|
||||
- **stacked** (459) — burst photos grouped into stacks
|
||||
|
||||
## Monitoring During Import
|
||||
|
||||
For background processes (via Hermes terminal background=true):
|
||||
|
||||
```python
|
||||
# Check progress with poll()
|
||||
process(action="poll", session_id="proc_xxx")
|
||||
|
||||
# When the imported finishes, capture the full output:
|
||||
process(action="log", session_id="proc_xxx")
|
||||
```
|
||||
|
||||
## Sample Job Queue After Large Import
|
||||
|
||||
After importing ~18,595 assets (16,401 new), expect queues like:
|
||||
|
||||
```
|
||||
🟢 thumbnailGeneration: 3 active, 890 waiting
|
||||
🟢 metadataExtraction: 5 active, 10587 waiting
|
||||
🟢 videoConversion: 1 active, 260 waiting
|
||||
🟢 smartSearch: 2 active, 2574 waiting
|
||||
🟢 faceDetection: 2 active, 6698 waiting
|
||||
🟢 facialRecognition: 1 active, 4358 waiting
|
||||
🟢 ocr: 1 active, 8421 waiting
|
||||
🟢 storageTemplateMigration: 1 active, 825 waiting
|
||||
```
|
||||
|
||||
Processing all jobs on a 4-core i5-6500T with 7GB RAM takes approximately:
|
||||
- Thumbnails + metadata: 2-4 hours
|
||||
- Smart search (CLIP), face rec, OCR: 12-24 hours each
|
||||
- Video conversion: 10-20 hours (if enabled)
|
||||
@@ -0,0 +1,147 @@
|
||||
---
|
||||
name: immich-import
|
||||
description: Import photos into Immich from Google Takeout, local folders, and other sources using immich-go CLI
|
||||
tags:
|
||||
- immich
|
||||
- google-takeout
|
||||
- photo-import
|
||||
- self-hosting
|
||||
- immich-go
|
||||
---
|
||||
|
||||
# Immich Photo Import
|
||||
|
||||
Import photos into a self-hosted Immich server. Covers Google Takeout imports (the most common case), plain folder uploads, and troubleshooting.
|
||||
|
||||
## When to load this skill
|
||||
|
||||
- User says "import photos to Immich", "Google Takeout", "immich-go", or "upload to Immich"
|
||||
- User is migrating from Google Photos to Immich
|
||||
- User has a folder of photos to bulk-upload to Immich
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Immich server URL (e.g. `http://192.168.50.150:2283`)
|
||||
- API key with sufficient permissions (needs at minimum `user.read` + upload rights)
|
||||
- `immich-go` binary installed (for large imports) or the Immich CLI inside the server container
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Verify immich-go is installed
|
||||
|
||||
```bash
|
||||
/usr/local/bin/immich-go version
|
||||
```
|
||||
|
||||
Install if needed (from [simulot/immich-go](https://github.com/simulot/immich-go) releases).
|
||||
|
||||
### 2. Get a valid API key
|
||||
|
||||
**Option A — Via Immich web UI:**
|
||||
- Settings → API Keys → Create New Key → name it → copy the key
|
||||
|
||||
**Option B — Via Immich API (if you know the password):**
|
||||
```bash
|
||||
# Login and create key
|
||||
TOKEN=$(curl -s -X POST http://<server>:2283/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email":"admin@example.com","password":"..."}' | python3 -c "import json,sys; print(json.load(sys.stdin).get('accessToken',''))")
|
||||
|
||||
curl -s -X POST http://<server>:2283/api/api-key \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-d '{"name":"immich-go-import"}'
|
||||
```
|
||||
|
||||
**Option C — Find admin email from DB:**
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c \
|
||||
'SELECT id, email, name, "isAdmin" FROM "user";'
|
||||
```
|
||||
|
||||
### 3. Google Takeout Import
|
||||
|
||||
#### 3a. Download the Takeout zips
|
||||
|
||||
If using Chrome in a KasmVNC container to download and Chrome kills the downloads:
|
||||
|
||||
**Fix Chrome Safe Browsing (kills large Takeout zips):**
|
||||
```bash
|
||||
docker exec -u 0 chrome sh -c 'cat > /etc/opt/chrome/policies/managed/download_safety.json << '\''EOF'\''
|
||||
{
|
||||
"DownloadRestrictions": 0,
|
||||
"SafeBrowsingEnabled": false,
|
||||
"SafeBrowsingProtectionForDownloadEnabled": false
|
||||
}
|
||||
EOF
|
||||
'
|
||||
# Kill Chrome so it restarts with the new policies
|
||||
docker exec -u 0 chrome pkill -f "chrome" || true
|
||||
```
|
||||
|
||||
#### 3b. Extract the zips
|
||||
|
||||
```bash
|
||||
# install unzip if missing
|
||||
sudo apt install -y unzip
|
||||
|
||||
mkdir -p /path/to/extracted
|
||||
for f in takeout-*.zip; do
|
||||
unzip -q -o "$f" -d extracted/
|
||||
done
|
||||
```
|
||||
|
||||
#### 3c. Import to Immich
|
||||
|
||||
```bash
|
||||
# From Google Takeout (preserves albums, metadata, people tags):
|
||||
/usr/local/bin/immich-go \
|
||||
--server=http://<server>:2283 \
|
||||
--api-key=<your-key> \
|
||||
upload from-google-photos /path/to/extracted/
|
||||
|
||||
# From a plain folder (no Takeout metadata):
|
||||
/usr/local/bin/immich-go \
|
||||
--server=http://<server>:2283 \
|
||||
--api-key=<your-key> \
|
||||
upload from-folder --recursive /path/to/folder/
|
||||
```
|
||||
|
||||
Using the Immich CLI inside the server container instead:
|
||||
```bash
|
||||
docker exec immich_server /usr/src/app/server/bin/immich \
|
||||
--url http://localhost:3001 \
|
||||
--key <your-key> \
|
||||
upload --recursive /path/
|
||||
```
|
||||
|
||||
### 4. Verify
|
||||
|
||||
After import, check Immich web UI for:
|
||||
- Albums created (if using `from-google-photos`)
|
||||
- Asset counts
|
||||
- Background jobs processing (thumbnails, face detection, etc.)
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Flag names**: immich-go uses `--api-key` and `--server` (double-dash long form). Single-dash `-key` or `-server` gets parsed as short flags (e.g. `-k` + `ey=...`) and fails silently or shows help.
|
||||
- **API key permissions**: immich-go checks several endpoints during startup. A key needs **all** of these permissions or it will fail with 403 at each check:
|
||||
- `user.read` — connection validation (`GET /api/users/me`)
|
||||
- `server.about` — server info check (`GET /api/server/about`)
|
||||
- `asset.statistics` — pre-upload stats (`GET /api/assets/statistics`)
|
||||
- `asset.write` + `asset.read` — actual upload + duplicate detection
|
||||
- `job.create` — pausing background jobs during upload (optional — skip with `--pause-immich-jobs=FALSE`)
|
||||
|
||||
**Best practice:** Create a fresh key and enable **all permissions**, or you'll hit a whack-a-mole of 403s one endpoint at a time.
|
||||
- **Chrome Safe Browsing**: Google Takeout zips can be 50GB+. Chrome flags them as "uncommon" and kills the download with `FILE_SECURITY_CHECK_FAILED` (reason 40, danger UNCOMMON). The `.crdownload` files will be orphaned — they will never complete. Fix: set managed policies to disable Safe Browsing for downloads, then restart Chrome.
|
||||
- **`unzip` not installed**: Check first. On Ubuntu/Debian: `sudo apt install unzip`.
|
||||
- **DB table names**: Immich PostgreSQL uses lowercase table names. The user table is `"user"`, not `"users"`. Column names use camelCase quoting (e.g. `"isAdmin"`, `"oauthId"`).
|
||||
**Key permissions vs --pause-immich-jobs flag:** If the API key lacks `job.create`, immich-go fails trying to pause background jobs. Workaround: add `--pause-immich-jobs=FALSE` to skip the job-pausing step:
|
||||
```bash
|
||||
/usr/local/bin/immich-go \\
|
||||
--server=http://<server>:2283 \\
|
||||
--api-key=<your-key> \\
|
||||
--pause-immich-jobs=FALSE \\
|
||||
upload from-google-photos /path/to/extracted/
|
||||
```
|
||||
This lets uploads proceed without admin-level job permissions. Thumbnails/face detection will process in the background normally.
|
||||
@@ -0,0 +1,164 @@
|
||||
# Immich Postgres Diagnostics (No API Key)
|
||||
|
||||
When you don't have an Immich API key handy, you can check server health, job progress,
|
||||
ML state, and config directly via the Postgres database inside the Immich Postgres container.
|
||||
|
||||
## Finding DB Credentials
|
||||
|
||||
The DB creds are set on the Postgres container at creation time. Get them from Docker:
|
||||
|
||||
```bash
|
||||
docker inspect immich_postgres --format '{{json .Config.Env}}' | python3 -c "
|
||||
import json,sys
|
||||
env = json.load(sys.stdin)
|
||||
for e in env:
|
||||
if 'POSTGRES' in e.upper() or 'DB_' in e:
|
||||
print(e)
|
||||
"
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
POSTGRES_PASSWORD=ab496a...17a2
|
||||
POSTGRES_USER=postgres
|
||||
POSTGRES_DB=immich
|
||||
```
|
||||
|
||||
Run queries with:
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c "SQL HERE"
|
||||
```
|
||||
|
||||
If password is required, pass via env:
|
||||
```bash
|
||||
docker exec -e PGPASSWORD=PASS immich_postgres psql -U postgres -d immich -c "SQL HERE"
|
||||
```
|
||||
|
||||
## Job Queue Progress (per-asset job status)
|
||||
|
||||
Immich v2.7+ tracks per-asset job completion in `asset_job_status`. This is NOT a
|
||||
traditional queue — it shows which assets have had each job type complete:
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
COUNT(*) as total_assets,
|
||||
COUNT(*) FILTER (WHERE ajs."facesRecognizedAt" IS NOT NULL) as faces_done,
|
||||
COUNT(*) FILTER (WHERE ajs."metadataExtractedAt" IS NOT NULL) as metadata_done,
|
||||
COUNT(*) FILTER (WHERE ajs."duplicatesDetectedAt" IS NOT NULL) as dedup_done,
|
||||
COUNT(*) FILTER (WHERE ajs."ocrAt" IS NOT NULL) as ocr_done
|
||||
FROM asset a
|
||||
LEFT JOIN asset_job_status ajs ON a.id = ajs."assetId";
|
||||
```
|
||||
|
||||
**Table schema** (`\d asset_job_status`):
|
||||
```
|
||||
assetId | uuid | not null
|
||||
facesRecognizedAt | timestamp with time zone|
|
||||
metadataExtractedAt | timestamp with time zone|
|
||||
duplicatesDetectedAt | timestamp with time zone|
|
||||
ocrAt | timestamp with time zone|
|
||||
```
|
||||
|
||||
## System Metadata
|
||||
|
||||
The `system_metadata` table holds Immich's key state as JSONB:
|
||||
|
||||
```sql
|
||||
SELECT * FROM system_metadata;
|
||||
```
|
||||
|
||||
Key entries:
|
||||
|
||||
| key | value (snippet) | purpose |
|
||||
|-----|-----------------|---------|
|
||||
| `system-config` | `{"ffmpeg": {"transcode":"disabled"}, "storageTemplate": {"enabled": true}}` | Server config |
|
||||
| `facial-recognition-state` | `{"lastRun": "2026-05-31T00:00:05.175Z"}` | Last ML face run |
|
||||
| `reverse-geocoding-state` | `{"lastUpdate": "...", "lastImportFileName": "cities500.txt"}` | Geo data freshness |
|
||||
| `version-check-state` | `{"checkedAt": "...", "releaseVersion": "v2.7.5"}` | Version tracking |
|
||||
| `memories-state` | `{"lastOnThisDayDate": "..."}` | Memories feature state |
|
||||
| `system-flags` | `{"mountChecks": {"thumbs": true, "upload": true, ...}}` | Mount health |
|
||||
| `admin-onboarding` | `{"isOnboarded": true}` | Admin setup status |
|
||||
|
||||
To extract just the ML config:
|
||||
```sql
|
||||
SELECT json_data -> 'machineLearning' as ml_config FROM system_metadata WHERE key='system-config';
|
||||
```
|
||||
|
||||
## Asset & People Counts
|
||||
|
||||
```sql
|
||||
SELECT COUNT(*) as total_assets FROM asset;
|
||||
SELECT COUNT(*) as total_albums FROM album;
|
||||
SELECT COUNT(*) as total_people FROM person;
|
||||
```
|
||||
|
||||
## ML Container GPU Check
|
||||
|
||||
The ML container uses ONNX Runtime (not PyTorch directly). Check GPU availability:
|
||||
|
||||
```bash
|
||||
docker exec immich_machine_learning python3 -c "
|
||||
import onnxruntime as ort
|
||||
print(f'ONNX Runtime: {ort.__version__}')
|
||||
print(f'Providers: {ort.get_available_providers()}')
|
||||
print(f'CUDA: {\"CUDAExecutionProvider\" in ort.get_available_providers()}')
|
||||
"
|
||||
```
|
||||
|
||||
Check the container's configured device mode:
|
||||
```bash
|
||||
docker inspect immich_machine_learning --format '{{json .Config.Env}}' | python3 -c "
|
||||
import json,sys
|
||||
env = json.load(sys.stdin)
|
||||
for e in env:
|
||||
if 'DEVICE' in e or 'GPU' in e or 'CUDA' in e:
|
||||
print(e)
|
||||
"
|
||||
```
|
||||
|
||||
Expected output for GPU-enabled:
|
||||
```
|
||||
DEVICE=cuda
|
||||
NVIDIA_VISIBLE_DEVICES=all
|
||||
CUDA_VERSION=12.2.2
|
||||
```
|
||||
|
||||
ONNX Runtime 1.24+ typically offers `TensorrtExecutionProvider`, `CUDAExecutionProvider`,
|
||||
`CPUExecutionProvider`. If `CUDAExecutionProvider` is in the list, GPU ML is active.
|
||||
|
||||
## Quick All-in-One Health Check
|
||||
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c "
|
||||
SELECT 'assets' as check, COUNT(*)::text FROM asset
|
||||
UNION ALL
|
||||
SELECT 'people', COUNT(*)::text FROM person
|
||||
UNION ALL
|
||||
SELECT 'albums', COUNT(*)::text FROM album
|
||||
UNION ALL
|
||||
SELECT 'faces_done', COUNT(*) FILTER (WHERE ajs.\"facesRecognizedAt\" IS NOT NULL)::text
|
||||
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
|
||||
UNION ALL
|
||||
SELECT 'metadata_done', COUNT(*) FILTER (WHERE ajs.\"metadataExtractedAt\" IS NOT NULL)::text
|
||||
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
|
||||
UNION ALL
|
||||
SELECT 'ocr_done', COUNT(*) FILTER (WHERE ajs.\"ocrAt\" IS NOT NULL)::text
|
||||
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
|
||||
UNION ALL
|
||||
SELECT 'face_last_run', value->>'lastRun' FROM system_metadata WHERE key='facial-recognition-state'
|
||||
UNION ALL
|
||||
SELECT 'version', value->>'releaseVersion' FROM system_metadata WHERE key='version-check-state'
|
||||
UNION ALL
|
||||
SELECT 'transcoding', CASE WHEN value->'ffmpeg'->>'transcode' = 'disabled' THEN 'disabled' ELSE 'enabled' END
|
||||
FROM system_metadata WHERE key='system-config';
|
||||
"
|
||||
```
|
||||
|
||||
## When to use Postgres diagnostics vs API
|
||||
|
||||
| Situation | Best approach |
|
||||
|-----------|---------------|
|
||||
| You have an API key | `/api/jobs` endpoint (richer data: active, waiting, failed, paused counts) |
|
||||
| No API key / lost access | Postgres queries above |
|
||||
| Checking ML container health | `docker exec` + ONNX Runtime check |
|
||||
| GPU detection | `docker inspect` ML container env vars |
|
||||
@@ -0,0 +1,188 @@
|
||||
# Pre-Caching Immich ML Models for GPU Acceleration
|
||||
|
||||
> How to manually download and cache CLIP + face detection models so the ML
|
||||
> container starts cleanly on GPU without the download→fail→clear→retry loop.
|
||||
|
||||
## When to use this
|
||||
|
||||
The Immich ML container (`immich-machine-learning`) downloads models from
|
||||
HuggingFace on first startup. If the download fails or the model load fails
|
||||
(disk space, VRAM contention, interrupted download), it enters an infinite
|
||||
loop:
|
||||
|
||||
```
|
||||
WARNING Failed to load visual model 'ViT-B-32__openai'. Clearing cache.
|
||||
WARNING Failed to load detection model 'buffalo_l'. Clearing cache.
|
||||
Downloading visual model 'ViT-B-32__openai' to /cache/clip/...
|
||||
Downloading detection model 'buffalo_l' to /cache/facial-recognition/...
|
||||
→ repeat ad infinitum
|
||||
```
|
||||
|
||||
The container stays `(unhealthy)` and never serves ML requests.
|
||||
|
||||
## Cache directory structure
|
||||
|
||||
Each model type has a fixed cache path determined by Immich's `InferenceModel`
|
||||
base class:
|
||||
|
||||
| Model | Task (value) | Type (value) | Full Path |
|
||||
|-------|-------------|--------------|-----------|
|
||||
| CLIP visual | `clip` | `visual` | `/cache/clip/ViT-B-32__openai/visual/model.onnx` |
|
||||
| CLIP textual | `clip` | `textual` | `/cache/clip/ViT-B-32__openai/textual/model.onnx` |
|
||||
| Face detection | `facial-recognition` | `detection` | `/cache/facial-recognition/buffalo_l/detection/model.onnx` |
|
||||
| Face recognition | `facial-recognition` | `recognition` | `/cache/facial-recognition/buffalo_l/recognition/model.onnx` |
|
||||
|
||||
The formula is:
|
||||
```
|
||||
settings.cache_folder / model_task.value / model_name / model_type.value / model.onnx
|
||||
```
|
||||
Where `settings.cache_folder` is `/cache` by default.
|
||||
|
||||
## Pre-caching procedure
|
||||
|
||||
Run from the host against the running ML container. The venv is at
|
||||
`/opt/venv/bin/activate` and has `huggingface_hub` available.
|
||||
|
||||
### 1. CLIP model (smart search)
|
||||
|
||||
```bash
|
||||
docker exec -i immich_machine_learning bash << 'SCRIPT'
|
||||
source /opt/venv/bin/activate
|
||||
python -c "
|
||||
from huggingface_hub import snapshot_download
|
||||
import os
|
||||
cache_dir = '/cache/clip/ViT-B-32__openai'
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
snapshot_download(
|
||||
'immich-app/ViT-B-32__openai',
|
||||
cache_dir=cache_dir,
|
||||
local_dir=cache_dir,
|
||||
ignore_patterns=['*.armnn', '*.rknn'],
|
||||
max_workers=2,
|
||||
)
|
||||
print(f'CLIP model cached: {os.path.getsize(cache_dir + \"/visual/model.onnx\")/1024/1024:.0f} MB')
|
||||
"
|
||||
SCRIPT
|
||||
```
|
||||
|
||||
### 2. Face detection + recognition model (buffalo_l)
|
||||
|
||||
```bash
|
||||
docker exec -i immich_machine_learning bash << 'SCRIPT'
|
||||
source /opt/venv/bin/activate
|
||||
python -c "
|
||||
from huggingface_hub import snapshot_download
|
||||
import os
|
||||
cache_dir = '/cache/facial-recognition/buffalo_l'
|
||||
os.makedirs(cache_dir, exist_ok=True)
|
||||
snapshot_download(
|
||||
'immich-app/buffalo_l',
|
||||
cache_dir=cache_dir,
|
||||
local_dir=cache_dir,
|
||||
ignore_patterns=['*.armnn', '*.rknn'],
|
||||
max_workers=2,
|
||||
)
|
||||
print(f'Detection ONNX: {os.path.getsize(cache_dir + \"/detection/model.onnx\")/1024/1024:.0f} MB')
|
||||
print(f'Recognition ONNX: {os.path.getsize(cache_dir + \"/recognition/model.onnx\")/1024/1024:.0f} MB')
|
||||
"
|
||||
SCRIPT
|
||||
```
|
||||
|
||||
### 3. Verify models load on GPU
|
||||
|
||||
```bash
|
||||
docker exec -i immich_machine_learning bash << 'SCRIPT'
|
||||
source /opt/venv/bin/activate
|
||||
python -c "
|
||||
from immich_ml.models import from_model_type
|
||||
from immich_ml.schemas import ModelTask, ModelType
|
||||
import time
|
||||
|
||||
for name, mt, task in [
|
||||
('ViT-B-32__openai', ModelType.VISUAL, ModelTask.SEARCH),
|
||||
('buffalo_l', ModelType.DETECTION, ModelTask.FACIAL_RECOGNITION),
|
||||
('buffalo_l', ModelType.RECOGNITION, ModelTask.FACIAL_RECOGNITION),
|
||||
]:
|
||||
m = from_model_type(name, mt, task)
|
||||
start = time.time()
|
||||
m.load()
|
||||
print(f'{name} {mt.value}: loaded in {time.time()-start:.1f}s')
|
||||
"
|
||||
SCRIPT
|
||||
```
|
||||
|
||||
Expected output:
|
||||
```
|
||||
ViT-B-32__openai visual: loaded in 0.5s
|
||||
buffalo_l detection: loaded in 0.0s
|
||||
buffalo_l recognition: loaded in 1.2s
|
||||
```
|
||||
|
||||
### 4. Restart ML container
|
||||
|
||||
Once models are cached, restart so the health check passes:
|
||||
|
||||
```bash
|
||||
docker compose restart immich-machine-learning
|
||||
```
|
||||
|
||||
After restart, verify the logs show clean startup with no download/load warnings:
|
||||
```bash
|
||||
docker logs immich_machine_learning --tail 10
|
||||
```
|
||||
|
||||
### 5. Re-trigger ML jobs
|
||||
|
||||
The jobs were interrupted by the container restart. Re-trigger them:
|
||||
|
||||
```bash
|
||||
# Login
|
||||
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"email":"admin@example.com","password":"your-password"}')
|
||||
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")
|
||||
|
||||
# Trigger each job
|
||||
for JOB in smartSearch faceDetection facialRecognition thumbnailGeneration; do
|
||||
curl -s -X PUT "http://localhost:2283/api/jobs/$JOB" \
|
||||
-H "Authorization: Bearer *** \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"command":"start"}'
|
||||
done
|
||||
```
|
||||
|
||||
## Verifying GPU is working
|
||||
|
||||
After re-triggering, monitor:
|
||||
|
||||
```bash
|
||||
# GPU memory usage — should jump from 11 MiB to 1200+ MiB
|
||||
nvidia-smi --query-gpu=memory.used,utilization.gpu,temperature.gpu --format=csv,noheader
|
||||
|
||||
# ML container logs should show ONNX Runtime using CUDA providers
|
||||
docker logs immich_machine_learning | grep -i "cuda\|provider"
|
||||
|
||||
# Immich server should show face detection activity
|
||||
docker logs immich_server --tail 20 | grep "PersonService\|Detected"
|
||||
```
|
||||
|
||||
Expected GPU profile during active processing (GTX 1050 Ti 4GB):
|
||||
- VRAM: ~1,200–1,400 MiB
|
||||
- Utilization: 1–40% (bursty, depends on queue depth)
|
||||
- Temp: 10–15°C above idle (e.g., 28°C → 40°C)
|
||||
|
||||
## Performance expectations
|
||||
|
||||
| Model | Load time | Inference | Notes |
|
||||
|-------|-----------|-----------|-------|
|
||||
| CLIP visual (335 MB) | ~0.8s | ~237ms | Batch size 1, output shape (1, 512) |
|
||||
| Face detection (16 MB) | ~0.0s | instant | Small model |
|
||||
| Face recognition (166 MB) | ~1.2s | varies | Depends on face count per image |
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Container has no `curl` or `ping`.** Use Python's `urllib.request` from the activated venv for network tests.
|
||||
- **Model downloads require HuggingFace hub access.** The `immich-app/*` repos are public but use `snapshot_download()`, not raw curl. Plain HTTPS requests return 401.
|
||||
- **Cache dir is a Docker volume.** If you recreate the container with `docker compose down -v`, the cache is lost and you need to pre-cache again.
|
||||
- **Pre-caching too many models may fill VRAM.** The CLIP model alone is 335 MB; buffalo_l adds ~182 MB. Total ~517 MB in VRAM, well within a 4 GB GPU. But loading ALL models simultaneously may cause OOM if other processes are using VRAM.
|
||||
- **Model arena (`MACHINE_LEARNING_MODEL_ARENA=true`)** can help if VRAM is tight by running models sequentially rather than keeping all in memory.
|
||||
@@ -0,0 +1,129 @@
|
||||
# Recovering Immich Admin Access via PostgreSQL
|
||||
|
||||
> Companion for the `immich-server` skill. Covers the "locked out of admin" scenario — looking up user info, resetting passwords, and understanding how API keys are stored. Assumes you have shell access to the Docker host where Immich runs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Shell access to the machine running Immich Docker containers
|
||||
- The `immich_postgres` container is running
|
||||
- The DB credentials from `/opt/immich/.env` (or wherever your compose lives)
|
||||
|
||||
## Find the Immich PostgreSQL Table Structure
|
||||
|
||||
The `user` table (note: lowercase, not `users`) stores everything:
|
||||
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c "\dt"
|
||||
```
|
||||
|
||||
Full list of tables (61+ as of Immich v2.7.x):
|
||||
|
||||
```
|
||||
activity, album, album_asset, album_user, api_key, asset, asset_exif, asset_face,
|
||||
asset_file, asset_job_status, asset_metadata, asset_ocr, face_search, library,
|
||||
memory, person, session, shared_link, smart_search, stack, tag, tag_asset,
|
||||
user, user_metadata, workflow, ...
|
||||
```
|
||||
|
||||
## Find Your Admin Email
|
||||
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c \
|
||||
"SELECT id, email, name, \"isAdmin\", \"oauthId\" FROM \"user\";"
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
id | email | name | isAdmin
|
||||
--------------------+---------------------------+------+---------
|
||||
523cab1b-... | you@gmail.com | You | t
|
||||
4defcb72-... | partner@gmail.com | Name | f
|
||||
```
|
||||
|
||||
- `isAdmin = t` = admin account
|
||||
- `isAdmin = f` = user account / partner share
|
||||
|
||||
## Reset an Admin Password
|
||||
|
||||
If you don't know the password and can't log in:
|
||||
|
||||
**Step 1 — Install bcrypt on the host:**
|
||||
|
||||
```bash
|
||||
sudo apt install -y python3-bcrypt
|
||||
```
|
||||
|
||||
**Step 2 — Generate a bcrypt hash for your new password:**
|
||||
|
||||
```bash
|
||||
HASH=$(python3 -c "
|
||||
import bcrypt
|
||||
# Use 'admin123' or whatever you want
|
||||
pw_hash = bcrypt.hashpw(b'admin123', bcrypt.gensalt(rounds=10))
|
||||
print(pw_hash.decode())
|
||||
")
|
||||
```
|
||||
|
||||
**Step 3 — Update the password in the database:**
|
||||
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c \
|
||||
"UPDATE \"user\" SET \"password\"='$HASH' WHERE email='you@gmail.com';"
|
||||
```
|
||||
|
||||
**Step 4 — Log in** with the new password at `http://192.168.x.x:2283`.
|
||||
|
||||
> ⚠️ The Immich server may cache the old session — refresh the login page or use a private browser window.
|
||||
|
||||
## API Keys: Storage & Recovery
|
||||
|
||||
**API keys are stored hashed (PBKDF2-SHA256) — you CANNOT recover the raw key from the DB.**
|
||||
|
||||
### List all API keys (who owns what)
|
||||
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -c \
|
||||
"SELECT ak.id, ak.name, u.name as owner, \"userId\" FROM \"api_key\" ak LEFT JOIN \"user\" u ON ak.\"userId\" = u.id;"
|
||||
```
|
||||
|
||||
The `key` column stores the hash — no way to reverse it. If you lost the raw key, you must:
|
||||
|
||||
1. Log in to Immich web UI as admin
|
||||
2. Go to **Settings → API Keys**
|
||||
3. Delete the old key and create a new one
|
||||
|
||||
### If you can't log in at all
|
||||
|
||||
Reset the admin password (above), then use the web UI to create a new API key.
|
||||
|
||||
## Finding the DB Credentials
|
||||
|
||||
Immich stores them in the compose directory:
|
||||
|
||||
```bash
|
||||
cat /opt/immich/.env
|
||||
```
|
||||
|
||||
Expected vars:
|
||||
```
|
||||
DB_USERNAME=postgres
|
||||
DB_PASSWORD=...
|
||||
DB_DATABASE_NAME=immich
|
||||
```
|
||||
|
||||
If the .env file path differs, find it:
|
||||
```bash
|
||||
find / -name "docker-compose.yml" -path "*immich*" 2>/dev/null
|
||||
```
|
||||
Then check the same directory for the .env file.
|
||||
|
||||
## Test DB Connection Directly
|
||||
|
||||
```bash
|
||||
# Simple auth test (no password prompt):
|
||||
docker exec immich_postgres psql -U postgres -d immich -c "SELECT 1 as connected;"
|
||||
|
||||
# If that fails with authentication error, try with password:
|
||||
docker exec -e PGPASSWORD=your_db_password immich_postgres psql \
|
||||
-U postgres -d immich -c "SELECT 1 as connected;"
|
||||
```
|
||||
@@ -0,0 +1,366 @@
|
||||
# YOLO GPU Classification on GTX 1050 Ti (Pascal, CC 6.1)
|
||||
|
||||
## CUDA/PyTorch Compatibility Matrix
|
||||
|
||||
| PyTorch | CUDA | Python | GTX 1050 Ti Result |
|
||||
|---------|------|--------|-------------------|
|
||||
| 2.12.0+cu130 | 13.0 | 3.14 | ❌ `no kernel image is available for execution on the device` |
|
||||
| 2.12.0+cu130 | 13.0 | 3.14 | ❌ `CUDA error: out of memory` (when zombie process holds VRAM) |
|
||||
| 2.5.1+cu124 | 12.4 | 3.12 | ✅ Working |
|
||||
|
||||
**Root cause:** PyTorch 2.12.0 (CUDA 13.0) dropped support for Pascal GPUs (compute capability 6.1) at the binary level — the shipped PTX/SASS targets newer architectures. The only PyTorch wheels available for Python 3.14 are 2.12.0+cu130, so **Python 3.12 must be used** with an older PyTorch + CUDA 12.4 build.
|
||||
|
||||
## Setup (Python 3.12 + CUDA 12.4)
|
||||
|
||||
```bash
|
||||
# Install Python 3.12 via deadsnakes (Ubuntu 26.04 doesn't ship it)
|
||||
sudo add-apt-repository -y ppa:deadsnakes/ppa
|
||||
sudo apt update
|
||||
sudo apt install -y python3.12 python3.12-venv
|
||||
|
||||
# Create venv with CUDA 12.4 PyTorch
|
||||
python3.12 -m venv /home/ray/yolo_venv_cu124
|
||||
source /home/ray/yolo_venv_cu124/bin/activate
|
||||
pip install torch==2.5.1+cu124 torchvision==0.20.1+cu124 --index-url https://download.pytorch.org/whl/cu124
|
||||
pip install ultralytics
|
||||
|
||||
# Verify (should show CUDA: True, CC: (6, 1))
|
||||
python3 -c 'import torch; print(f"CUDA: {torch.cuda.is_available()}, CC: {torch.cuda.get_device_capability(0)}")'
|
||||
```
|
||||
|
||||
**Why Python 3.12?** Ubuntu 26.04 ships Python 3.14.4 as default. PyTorch 2.5.x has no wheels for Python 3.14 — only 2.12.0+cu130, which doesn't support Pascal GPUs. Python 3.12 is the newest Python with CUDA 12.4 PyTorch wheels available via deadsnakes PPA.
|
||||
|
||||
## 🚨 Pitfalls
|
||||
|
||||
### Zombie GPU Processes After Background Kill
|
||||
|
||||
When Hermes kills a background process (e.g. `process(action="kill")`), the SSH session dies but the Python process on the remote server **keeps running and holds GPU VRAM**. This causes subsequent runs to OOM even with a compatible PyTorch version.
|
||||
|
||||
**Always check before launching a new GPU run:**
|
||||
|
||||
```bash
|
||||
# Check for zombies
|
||||
nvidia-smi
|
||||
# Look for python3 processes in the "Processes" section
|
||||
|
||||
# Kill all yolo processes
|
||||
pkill -f 'seagate_photos_yolo' # or your script name
|
||||
# Wait 1-2 seconds for GPU memory to release
|
||||
sleep 2
|
||||
nvidia-smi # Verify 0 MiB used
|
||||
```
|
||||
|
||||
**Symptom:** `CUDA error: out of memory` on a fresh run even though the script should fit in VRAM.
|
||||
|
||||
### Corrupt Image Fallback: Don't Retry on GPU
|
||||
|
||||
When a batch of images fails YOLO processing (corrupt JPEGs, truncated files), **do NOT retry each file individually on GPU**. The individual fallback path (`model([filepath], imgsz=384, ...)`) is:
|
||||
- Slow (model re-inference per file)
|
||||
- Unreliable (corrupt file still fails at lower resolution)
|
||||
- Wastes GPU cycles on unreadable data
|
||||
|
||||
**Instead:** On batch failure, move the entire batch directly to the target folder without GPU retry:
|
||||
|
||||
```python
|
||||
except Exception as e:
|
||||
# Batch-level failure — likely corrupt files
|
||||
for filepath in batch:
|
||||
try:
|
||||
rel_path = os.path.relpath(filepath, SRC)
|
||||
dst_path = os.path.join(DST, rel_path)
|
||||
os.makedirs(os.path.dirname(dst_path), exist_ok=True)
|
||||
shutil.move(filepath, dst_path)
|
||||
moved += 1
|
||||
except Exception:
|
||||
errors += 1
|
||||
continue
|
||||
```
|
||||
|
||||
### Background Output Buffering
|
||||
|
||||
Hermes' `process(action="poll")` may not show stdout from background SSH sessions due to pipe buffering. **Redirect output to a log file and read it with `read_file`** instead:
|
||||
|
||||
```bash
|
||||
# In background command:
|
||||
python3 -u /tmp/script.py > /tmp/script.log 2>&1
|
||||
|
||||
# Monitor progress:
|
||||
tail -3 /tmp/script.log
|
||||
# Or use read_file tool
|
||||
```
|
||||
|
||||
Use `python3 -u` (unbuffered) and `flush=True` on all print() calls.
|
||||
|
||||
## Full Working Script (Production)
|
||||
|
||||
The production script `yolo_gpu_immediate.py` (deployed at `/tmp/yolo_gpu_immediate.py` on rayserver) combines:
|
||||
|
||||
- Immich DB query via `docker exec immich_postgres psql ...`
|
||||
- Path mapping from container `/data/library/...` to real `/mnt/wd-passport/immich/photos/library/...`
|
||||
- YOLOv8n single-image GPU inference at ~6-7 img/s on GTX 1050 Ti
|
||||
- **Immediate `shutil.move()` per image** — not batched at end (user preference)
|
||||
- **`stream=True`** — prevents Ultralytics RAM accumulation warning
|
||||
- Progress logging every 500 images
|
||||
|
||||
### Querying Immich's PostgreSQL
|
||||
|
||||
```sql
|
||||
SELECT "originalPath" FROM asset
|
||||
WHERE type='IMAGE' AND visibility='timeline'
|
||||
AND "originalPath" LIKE '/data/library/%'
|
||||
ORDER BY "originalPath";
|
||||
```
|
||||
|
||||
Executed via:
|
||||
```bash
|
||||
docker exec immich_postgres psql -U postgres -d immich -t -A -c "<SQL>"
|
||||
```
|
||||
|
||||
The `"originalPath"` from Immich looks like `/data/library/4defcb72-1058-4fcd-826a-e2e87e59aa4c/2026/2026-05-11/...`. To map to real filesystem, strip the `/data/` prefix and prepend the real photo base (`/mnt/wd-passport/immich/photos`).
|
||||
|
||||
### Classification Ruleset
|
||||
|
||||
- **Person** (class 0) → keep
|
||||
- **Nature** (classes 16-27=birds/cats/dogs/horses/sheep/cow/elephant/bear/zebra/giraffe, 58=potted plant, 77=vase, 80=umbrella) → keep
|
||||
- **Urban/indoor move-ables** (classes 1-15 = human artifacts like car/bicycle/motorcycle/airplane/bus/train/truck/boat/traffic light/fire hydrant/stop sign/parking meter/bench, 56=chair, 57=couch, 59=bed, 60=dining table, 61=toilet, 62=tv, 63=laptop, 64=mouse, 65=remote, 67=cell phone, 70=oven, 71=toaster, 72=sink, 73=refrigerator, 74=book, 75=clock, 76=keyboard, 78=scissors, 79=teddy bear) → move
|
||||
- **No detections + file <400KB** → move (screenshots, memes)
|
||||
- **No detections + file >=400KB** → keep (likely scenic with no recognizable objects)
|
||||
|
||||
### GPU Memory Profile
|
||||
|
||||
During single-image inference on GTX 1050 Ti (4GB):
|
||||
- Memory used: ~445 MB
|
||||
- GPU utilization: ~5% (bottlenecked by image loading/preprocessing, not compute)
|
||||
- Batch size 4 can increase utilization but risks OOM for larger images
|
||||
|
||||
## Background Execution (Keep Agent Responsive)
|
||||
|
||||
The user explicitly wants all long-running tasks in background so Hermes stays accessible.
|
||||
|
||||
**Pattern: redirect to log file, monitor with read_file.**
|
||||
|
||||
```bash
|
||||
# Start (via Hermes terminal background=true)
|
||||
ssh rayserver "source /home/ray/yolo_venv_cu124/bin/activate && python3 -u /tmp/script.py > /tmp/yolo_seagate.log 2>&1"
|
||||
|
||||
# Monitor progress — use read_file, NOT process poll (buffering issue)
|
||||
read_file /tmp/yolo_seagate.log
|
||||
|
||||
# Files moved so far
|
||||
find /mnt/seagate8tb/NO\ PEOPLE/ -type f | wc -l
|
||||
|
||||
# Kill zombies (check nvidia-smi first!)
|
||||
pkill -f script_name
|
||||
nvidia-smi # verify GPU memory freed
|
||||
```
|
||||
|
||||
**Never use `tail` via terminal for log monitoring** — use `read_file` instead. Background SSH stdout is buffered at the pipe level and `process(action="poll")` often shows empty output even when the script is actively writing.
|
||||
|
||||
## EXIF Rotation: Critical Fix for Phone Photos
|
||||
|
||||
**YOLO via OpenCV ignores EXIF orientation.** Phone portrait photos (which are stored landscape with an EXIF rotation flag) will appear sideways to the model. People in these photos are routinely missed.
|
||||
|
||||
### Fix: PIL load with EXIF transpose
|
||||
|
||||
```python
|
||||
from PIL import Image, ImageOps
|
||||
import numpy as np
|
||||
|
||||
with Image.open(ap) as img:
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img is None:
|
||||
img = Image.open(ap) # reload if exif_transpose returned None
|
||||
if img.mode != 'RGB':
|
||||
img = img.convert('RGB') # handle RGBA (PNG alpha channel), CMYK, etc.
|
||||
img_np = np.array(img)[:, :, ::-1] # RGB -> BGR for Ultralytics pipeline
|
||||
results = model(img_np, device="cuda:0", verbose=False)
|
||||
```
|
||||
|
||||
**Important:** After `exif_transpose`, the image MUST be converted to `'RGB'` mode. PNGs with alpha channels are `RGBA` (4 channels) and YOLO expects exactly 3.
|
||||
|
||||
### 4-Orientation Brute-Force (When EXIF Alone Fails)
|
||||
|
||||
Some photos are stored rotated in pixel data WITHOUT any EXIF orientation flag. `ImageOps.exif_transpose()` returns them unchanged. Fix: try all 4 rotations:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
# After PIL load (RGB), convert to BGR for YOLO pipeline
|
||||
img_bgr = np.array(img)[:, :, ::-1]
|
||||
|
||||
orientations = {
|
||||
'0°': img_bgr,
|
||||
'90°': np.rot90(img_bgr, k=3), # 90° CW
|
||||
'180°': np.rot90(img_bgr, k=2), # 180°
|
||||
'270°': np.rot90(img_bgr, k=1), # 270° CW
|
||||
}
|
||||
|
||||
found_person = False
|
||||
for orient_name, orient_img in orientations.items():
|
||||
results = model(orient_img, device="cuda:0", verbose=False)
|
||||
for r in results:
|
||||
if r.boxes:
|
||||
for box in r.boxes:
|
||||
if int(box.cls[0]) == 0: # person class
|
||||
found_person = True
|
||||
break
|
||||
if found_person: break
|
||||
if found_person: break
|
||||
```
|
||||
|
||||
**Performance impact:** 4x slower (~2.5-3 effective img/s on GTX 1050 Ti). Run as background task. Typically restores 2-7% of flagged photos as having people.
|
||||
|
||||
### PIL Decompression Bomb Protection
|
||||
|
||||
Pillow refuses to load images larger than 178MP by default (decompression bomb safety). Photos from phone panoramas or photo stitches (200MP+) will fail with:
|
||||
```
|
||||
Image size (199756800 pixels) exceeds limit of 178956970 pixels
|
||||
```
|
||||
|
||||
These images cannot be checked via PIL. Two options:
|
||||
- **Skip them** (they stay in their current classification, usually with a count in the error tally)
|
||||
- **Raise the limit** with `Image.MAX_IMAGE_PIXELS = None` (not recommended — can cause OOM on 4GB GPU)
|
||||
|
||||
## Post-Classification Sorting: Trash vs Keep
|
||||
|
||||
After moving non-people photos to a separate folder, use YOLO's object detections to categorize them into Trash (likely insignificant) and Keep (potentially important).
|
||||
|
||||
### Classification Rules
|
||||
|
||||
**TRASH** (screenshots, household clutter, no meaningful content):
|
||||
- cell phone (67), tv (62), laptop (63), keyboard (66), mouse (64), remote (65)
|
||||
- toilet (61), chair (56), couch (57), refrigerator (72), microwave (68), oven (69), sink (71)
|
||||
- hair drier (78), toothbrush (79), scissors (76)
|
||||
- No detections at all (blank/solid/text-only images)
|
||||
|
||||
**KEEP** (potentially important):
|
||||
- book (73) — documents, receipts, textbooks
|
||||
- car (2), truck (7), bus (5), motorcycle (3), bicycle (1), airplane (4) — vehicles/travel
|
||||
- dining table (60), cup (41), bottle (39), bowl (45), wine glass (40) — food/meals
|
||||
- dog (16), cat (15), bird (14) — pets
|
||||
- sports ball (32), cake (55), pizza (53) — events/food
|
||||
- clock (74) — time-stamped events
|
||||
- potted plant (58), vase (75), backpack (24), umbrella (25) — personal/meaningful
|
||||
|
||||
A photo with ANY "Keep" class detected goes to Keep. ONLY photos where ALL detected classes are Trash go to Trash. No-detection photos go to Trash.
|
||||
|
||||
### Typical Distribution
|
||||
|
||||
On a real-world photo library (16K photos), after YOLOv8n classification:
|
||||
- **~24%** flagged for removal (no people, no scenic content)
|
||||
- **~22%** of that flagged set is screenshots/trash
|
||||
- **~55%** is documents/books
|
||||
- **~15%** is vehicles/meals/personal worth keeping
|
||||
- **~3%** has missed people in rotated orientation (restored via multi-orientation check)
|
||||
|
||||
## Restore Pass: Re-Checking Flagged Photos
|
||||
|
||||
After the initial classification pass, ALWAYS do a restore pass on the moved photos. The user will spot false positives. The restore workflow:
|
||||
|
||||
1. Scan all files in the NO PEOPLE folder
|
||||
2. For each, load with PIL + EXIF transpose + 4 orientations
|
||||
3. If ANY orientation detects a person (class 0), `shutil.move()` back to original path
|
||||
4. Log which orientation caught it (helps confirm the fix works)
|
||||
|
||||
```python
|
||||
# Restore path calculation
|
||||
orig = ap.replace(NO_PEOPLE_DIR, PHOTO_BASE)
|
||||
os.makedirs(os.path.dirname(orig), exist_ok=True)
|
||||
shutil.move(ap, orig)
|
||||
```
|
||||
|
||||
## Flattening Moved Photos
|
||||
|
||||
After moving flagged photos out of Immich's structured directory tree, flatten them into a single folder for easier browsing:
|
||||
|
||||
```bash
|
||||
mkdir -p "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos"
|
||||
find "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/" -type f -not -path '*/ALL Photos/*' -exec mv -t "/mnt/wd-passport/immich/NO PEOPLE PHOTOS/ALL Photos/" {} +
|
||||
```
|
||||
|
||||
Handle filename collisions with number suffixes.
|
||||
|
||||
## Full Pipeline Workflow Order
|
||||
|
||||
The complete YOLO classification workflow runs in this order:
|
||||
|
||||
```
|
||||
1. CLASSIFY → 2. RESTORE → 3. FLATTEN → 4. SORT
|
||||
(initial) (re-check) (remove (trash vs keep
|
||||
4-orient) subdirs) by detection)
|
||||
```
|
||||
|
||||
### Step 1: Classify
|
||||
- Query Immich DB for all image paths
|
||||
- Map container paths to real filesystem paths
|
||||
- Run YOLOv8n on GPU, immediate `shutil.move()` on non-people/non-scenic photos
|
||||
- Log progress every 500 images
|
||||
|
||||
### Step 2: Restore (Multi-Orientation Re-check)
|
||||
- Scan all moved files in the NO PEOPLE target folder
|
||||
- For each, load with PIL + EXIF transpose + try all 4 orientations
|
||||
- If ANY orientation detects a person (class 0), move back to original path
|
||||
- Log which orientation caught the detection
|
||||
- Typically restores 2-7% of flagged photos
|
||||
|
||||
### Step 3: Flatten
|
||||
- After restore pass, gather all remaining photos into a single flat directory
|
||||
- Remove the deep Immich subdirectory structure (UUID/YYYY/YYYY-MM-DD/)
|
||||
- Handle filename collisions with `_1`, `_2` suffixes
|
||||
- Result: one folder with N files, no subdirectories
|
||||
|
||||
### Step 4: Sort (Trash vs Keep)
|
||||
- Run YOLO on all flattened photos to detect what objects are present
|
||||
- Categorize each photo:
|
||||
- **Trash**: ALL detected classes are screenshots/clutter AND no meaningful detections
|
||||
- **Keep**: ANY detected class is a "keep" class (vehicles, food, pets, documents, etc.)
|
||||
- Move into Trash/ and Keep/ subfolders
|
||||
- Photos with no detections → Trash
|
||||
|
||||
### Flatten Script (with Collision Handling)
|
||||
|
||||
```python
|
||||
import os, shutil
|
||||
|
||||
SRC = "/path/to/NO PEOPLE PHOTOS"
|
||||
DST = os.path.join(SRC, "ALL Photos")
|
||||
os.makedirs(DST, exist_ok=True)
|
||||
|
||||
files = []
|
||||
for root, dirs, fnames in os.walk(SRC):
|
||||
if root.startswith(DST):
|
||||
continue
|
||||
for f in fnames:
|
||||
files.append(os.path.join(root, f))
|
||||
|
||||
collisions = 0
|
||||
for src in files:
|
||||
fname = os.path.basename(src)
|
||||
dst = os.path.join(DST, fname)
|
||||
if os.path.exists(dst):
|
||||
base, ext = os.path.splitext(fname)
|
||||
n = 1
|
||||
while os.path.exists(os.path.join(DST, f"{base}_{n}{ext}")):
|
||||
n += 1
|
||||
dst = os.path.join(DST, f"{base}_{n}{ext}")
|
||||
collisions += 1
|
||||
shutil.move(src, dst)
|
||||
|
||||
print(f"Moved {len(files)} files, {collisions} renamed")
|
||||
```
|
||||
|
||||
### Sort Script (Trash vs Keep)
|
||||
|
||||
```python
|
||||
import os, shutil, csv
|
||||
|
||||
# See reference above for TRASH_CLASSES set definition
|
||||
# See the Post-Classification Sorting section above for the detailed class lists
|
||||
# CSV file produced by the categorization run: filename,detected_classes
|
||||
# A photo goes to KEEP if ANY detected class is NOT in the TRASH set
|
||||
# Otherwise it goes to TRASH (which also catches no-detection photos)
|
||||
```
|
||||
|
||||
## Full Pipeline Script Template
|
||||
|
||||
For a complete end-to-end script combining classification, EXIF rotation, 4-orientation restore, and sorting, combine the patterns above. Run as background process with nohup. The user expects to check progress periodically and be able to interact with the agent while the job runs.
|
||||
Reference in New Issue
Block a user