ray/hermes-config
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
ray
2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/software-development/immich-server/references/pre-caching-ml-models.md
+188
View File
@@ -0,0 +1,188 @@
# Pre-Caching Immich ML Models for GPU Acceleration
> How to manually download and cache CLIP + face detection models so the ML
> container starts cleanly on GPU without the download→fail→clear→retry loop.
## When to use this
The Immich ML container (`immich-machine-learning`) downloads models from
HuggingFace on first startup. If the download fails or the model load fails
(disk space, VRAM contention, interrupted download), it enters an infinite
loop:
```
WARNING Failed to load visual model 'ViT-B-32__openai'. Clearing cache.
WARNING Failed to load detection model 'buffalo_l'. Clearing cache.
Downloading visual model 'ViT-B-32__openai' to /cache/clip/...
Downloading detection model 'buffalo_l' to /cache/facial-recognition/...
→ repeat ad infinitum
```
The container stays `(unhealthy)` and never serves ML requests.
## Cache directory structure
Each model type has a fixed cache path determined by Immich's `InferenceModel`
base class:
| Model | Task (value) | Type (value) | Full Path |
|-------|-------------|--------------|-----------|
| CLIP visual | `clip` | `visual` | `/cache/clip/ViT-B-32__openai/visual/model.onnx` |
| CLIP textual | `clip` | `textual` | `/cache/clip/ViT-B-32__openai/textual/model.onnx` |
| Face detection | `facial-recognition` | `detection` | `/cache/facial-recognition/buffalo_l/detection/model.onnx` |
| Face recognition | `facial-recognition` | `recognition` | `/cache/facial-recognition/buffalo_l/recognition/model.onnx` |
The formula is:
```
settings.cache_folder / model_task.value / model_name / model_type.value / model.onnx
```
Where `settings.cache_folder` is `/cache` by default.
## Pre-caching procedure
Run from the host against the running ML container. The venv is at
`/opt/venv/bin/activate` and has `huggingface_hub` available.
### 1. CLIP model (smart search)
```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from huggingface_hub import snapshot_download
import os
cache_dir = '/cache/clip/ViT-B-32__openai'
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(
'immich-app/ViT-B-32__openai',
cache_dir=cache_dir,
local_dir=cache_dir,
ignore_patterns=['*.armnn', '*.rknn'],
max_workers=2,
)
print(f'CLIP model cached: {os.path.getsize(cache_dir + \"/visual/model.onnx\")/1024/1024:.0f} MB')
"
SCRIPT
```
### 2. Face detection + recognition model (buffalo_l)
```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from huggingface_hub import snapshot_download
import os
cache_dir = '/cache/facial-recognition/buffalo_l'
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(
'immich-app/buffalo_l',
cache_dir=cache_dir,
local_dir=cache_dir,
ignore_patterns=['*.armnn', '*.rknn'],
max_workers=2,
)
print(f'Detection ONNX: {os.path.getsize(cache_dir + \"/detection/model.onnx\")/1024/1024:.0f} MB')
print(f'Recognition ONNX: {os.path.getsize(cache_dir + \"/recognition/model.onnx\")/1024/1024:.0f} MB')
"
SCRIPT
```
### 3. Verify models load on GPU
```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from immich_ml.models import from_model_type
from immich_ml.schemas import ModelTask, ModelType
import time
for name, mt, task in [
('ViT-B-32__openai', ModelType.VISUAL, ModelTask.SEARCH),
('buffalo_l', ModelType.DETECTION, ModelTask.FACIAL_RECOGNITION),
('buffalo_l', ModelType.RECOGNITION, ModelTask.FACIAL_RECOGNITION),
]:
m = from_model_type(name, mt, task)
start = time.time()
m.load()
print(f'{name} {mt.value}: loaded in {time.time()-start:.1f}s')
"
SCRIPT
```
Expected output:
```
ViT-B-32__openai visual: loaded in 0.5s
buffalo_l detection: loaded in 0.0s
buffalo_l recognition: loaded in 1.2s
```
### 4. Restart ML container
Once models are cached, restart so the health check passes:
```bash
docker compose restart immich-machine-learning
```
After restart, verify the logs show clean startup with no download/load warnings:
```bash
docker logs immich_machine_learning --tail 10
```
### 5. Re-trigger ML jobs
The jobs were interrupted by the container restart. Re-trigger them:
```bash
# Login
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
-H "Content-Type: application/json" \
-d '{"email":"admin@example.com","password":"your-password"}')
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")
# Trigger each job
for JOB in smartSearch faceDetection facialRecognition thumbnailGeneration; do
curl -s -X PUT "http://localhost:2283/api/jobs/$JOB" \
-H "Authorization: Bearer *** \
-H "Content-Type: application/json" \
-d '{"command":"start"}'
done
```
## Verifying GPU is working
After re-triggering, monitor:
```bash
# GPU memory usage — should jump from 11 MiB to 1200+ MiB
nvidia-smi --query-gpu=memory.used,utilization.gpu,temperature.gpu --format=csv,noheader
# ML container logs should show ONNX Runtime using CUDA providers
docker logs immich_machine_learning | grep -i "cuda\|provider"
# Immich server should show face detection activity
docker logs immich_server --tail 20 | grep "PersonService\|Detected"
```
Expected GPU profile during active processing (GTX 1050 Ti 4GB):
- VRAM: ~1,200–1,400 MiB
- Utilization: 1–40% (bursty, depends on queue depth)
- Temp: 10–15°C above idle (e.g., 28°C → 40°C)
## Performance expectations
| Model | Load time | Inference | Notes |
|-------|-----------|-----------|-------|
| CLIP visual (335 MB) | ~0.8s | ~237ms | Batch size 1, output shape (1, 512) |
| Face detection (16 MB) | ~0.0s | instant | Small model |
| Face recognition (166 MB) | ~1.2s | varies | Depends on face count per image |
## Pitfalls
- **Container has no `curl` or `ping`.** Use Python's `urllib.request` from the activated venv for network tests.
- **Model downloads require HuggingFace hub access.** The `immich-app/*` repos are public but use `snapshot_download()`, not raw curl. Plain HTTPS requests return 401.
- **Cache dir is a Docker volume.** If you recreate the container with `docker compose down -v`, the cache is lost and you need to pre-cache again.
- **Pre-caching too many models may fill VRAM.** The CLIP model alone is 335 MB; buffalo_l adds ~182 MB. Total ~517 MB in VRAM, well within a 4 GB GPU. But loading ALL models simultaneously may cause OOM if other processes are using VRAM.
- **Model arena (`MACHINE_LEARNING_MODEL_ARENA=true`)** can help if VRAM is tight by running models sequentially rather than keeping all in memory.