hermes-config/skills/software-development/immich-server/references/pre-caching-ml-models.md

# Pre-Caching Immich ML Models for GPU Acceleration

> How to manually download and cache CLIP + face detection models so the ML
> container starts cleanly on GPU without the download→fail→clear→retry loop.

## When to use this

The Immich ML container (`immich-machine-learning`) downloads models from
HuggingFace on first startup. If the download fails or the model load fails
(disk space, VRAM contention, interrupted download), it enters an infinite
loop:

```
WARNING  Failed to load visual model 'ViT-B-32__openai'. Clearing cache.
WARNING  Failed to load detection model 'buffalo_l'. Clearing cache.
Downloading visual model 'ViT-B-32__openai' to /cache/clip/...
Downloading detection model 'buffalo_l' to /cache/facial-recognition/...
→ repeat ad infinitum
```

The container stays `(unhealthy)` and never serves ML requests.

## Cache directory structure

Each model type has a fixed cache path determined by Immich's `InferenceModel`
base class:

| Model | Task (value) | Type (value) | Full Path |
|-------|-------------|--------------|-----------|
| CLIP visual | `clip` | `visual` | `/cache/clip/ViT-B-32__openai/visual/model.onnx` |
| CLIP textual | `clip` | `textual` | `/cache/clip/ViT-B-32__openai/textual/model.onnx` |
| Face detection | `facial-recognition` | `detection` | `/cache/facial-recognition/buffalo_l/detection/model.onnx` |
| Face recognition | `facial-recognition` | `recognition` | `/cache/facial-recognition/buffalo_l/recognition/model.onnx` |

The formula is:
```
settings.cache_folder / model_task.value / model_name / model_type.value / model.onnx
```
Where `settings.cache_folder` is `/cache` by default.

## Pre-caching procedure

Run from the host against the running ML container. The venv is at
`/opt/venv/bin/activate` and has `huggingface_hub` available.

### 1. CLIP model (smart search)

```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from huggingface_hub import snapshot_download
import os
cache_dir = '/cache/clip/ViT-B-32__openai'
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(
    'immich-app/ViT-B-32__openai',
    cache_dir=cache_dir,
    local_dir=cache_dir,
    ignore_patterns=['*.armnn', '*.rknn'],
    max_workers=2,
)
print(f'CLIP model cached: {os.path.getsize(cache_dir + \"/visual/model.onnx\")/1024/1024:.0f} MB')
"
SCRIPT
```

### 2. Face detection + recognition model (buffalo_l)

```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from huggingface_hub import snapshot_download
import os
cache_dir = '/cache/facial-recognition/buffalo_l'
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(
    'immich-app/buffalo_l',
    cache_dir=cache_dir,
    local_dir=cache_dir,
    ignore_patterns=['*.armnn', '*.rknn'],
    max_workers=2,
)
print(f'Detection ONNX: {os.path.getsize(cache_dir + \"/detection/model.onnx\")/1024/1024:.0f} MB')
print(f'Recognition ONNX: {os.path.getsize(cache_dir + \"/recognition/model.onnx\")/1024/1024:.0f} MB')
"
SCRIPT
```

### 3. Verify models load on GPU

```bash
docker exec -i immich_machine_learning bash << 'SCRIPT'
source /opt/venv/bin/activate
python -c "
from immich_ml.models import from_model_type
from immich_ml.schemas import ModelTask, ModelType
import time

for name, mt, task in [
    ('ViT-B-32__openai', ModelType.VISUAL, ModelTask.SEARCH),
    ('buffalo_l', ModelType.DETECTION, ModelTask.FACIAL_RECOGNITION),
    ('buffalo_l', ModelType.RECOGNITION, ModelTask.FACIAL_RECOGNITION),
]:
    m = from_model_type(name, mt, task)
    start = time.time()
    m.load()
    print(f'{name} {mt.value}: loaded in {time.time()-start:.1f}s')
"
SCRIPT
```

Expected output:
```
ViT-B-32__openai visual: loaded in 0.5s
buffalo_l detection: loaded in 0.0s
buffalo_l recognition: loaded in 1.2s
```

### 4. Restart ML container

Once models are cached, restart so the health check passes:

```bash
docker compose restart immich-machine-learning
```

After restart, verify the logs show clean startup with no download/load warnings:
```bash
docker logs immich_machine_learning --tail 10
```

### 5. Re-trigger ML jobs

The jobs were interrupted by the container restart. Re-trigger them:

```bash
# Login
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"your-password"}')
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")

# Trigger each job
for JOB in smartSearch faceDetection facialRecognition thumbnailGeneration; do
  curl -s -X PUT "http://localhost:2283/api/jobs/$JOB" \
    -H "Authorization: Bearer *** \
    -H "Content-Type: application/json" \
    -d '{"command":"start"}'
done
```

## Verifying GPU is working

After re-triggering, monitor:

```bash
# GPU memory usage — should jump from 11 MiB to 1200+ MiB
nvidia-smi --query-gpu=memory.used,utilization.gpu,temperature.gpu --format=csv,noheader

# ML container logs should show ONNX Runtime using CUDA providers
docker logs immich_machine_learning | grep -i "cuda\|provider"

# Immich server should show face detection activity
docker logs immich_server --tail 20 | grep "PersonService\|Detected"
```

Expected GPU profile during active processing (GTX 1050 Ti 4GB):
- VRAM: ~1,200–1,400 MiB
- Utilization: 1–40% (bursty, depends on queue depth)
- Temp: 10–15°C above idle (e.g., 28°C → 40°C)

## Performance expectations

| Model | Load time | Inference | Notes |
|-------|-----------|-----------|-------|
| CLIP visual (335 MB) | ~0.8s | ~237ms | Batch size 1, output shape (1, 512) |
| Face detection (16 MB) | ~0.0s | instant | Small model |
| Face recognition (166 MB) | ~1.2s | varies | Depends on face count per image |

## Pitfalls

- **Container has no `curl` or `ping`.** Use Python's `urllib.request` from the activated venv for network tests.
- **Model downloads require HuggingFace hub access.** The `immich-app/*` repos are public but use `snapshot_download()`, not raw curl. Plain HTTPS requests return 401.
- **Cache dir is a Docker volume.** If you recreate the container with `docker compose down -v`, the cache is lost and you need to pre-cache again.
- **Pre-caching too many models may fill VRAM.** The CLIP model alone is 335 MB; buffalo_l adds ~182 MB. Total ~517 MB in VRAM, well within a 4 GB GPU. But loading ALL models simultaneously may cause OOM if other processes are using VRAM.
- **Model arena (`MACHINE_LEARNING_MODEL_ARENA=true`)** can help if VRAM is tight by running models sequentially rather than keeping all in memory.