hermes-config/skills/software-development/immich-server/references/immich-api-jobs.md

# Immich API: Job Management & GPU Troubleshooting

> Session-specific commands for triggering ML jobs and diagnosing GPU acceleration on the Immich machine-learning container (v2.7.5+).

## Authentication

### Login (get bearer token)

```bash
LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"your-password"}')
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")
```

The token is a bearer token, used as `Authorization: Bearer *** all subsequent API calls.

## Job Management

### List all job queues

```bash
curl -s http://localhost:2283/api/jobs -H "Authorization: Bearer ***
```

Returns each job name with `queueStatus` (isPaused, isActive) and `jobCounts` (active, completed, failed, delayed, waiting, paused).

### Trigger a specific job

```bash
curl -s -X PUT "http://localhost:2283/api/jobs/smartSearch" \
  -H "Authorization: Bearer *** \
  -H "Content-Type: application/json" \
  -d '{"command":"start"}'
```

To force re-process already-completed assets, use `{"command":"start","force":true}`.

### Available job names

- `smartSearch` — CLIP embeddings for natural language search
- `faceDetection` — detect faces in photos
- `facialRecognition` — group detected faces into people
- `metadataExtraction` — read EXIF dates, GPS, camera info
- `thumbnailGeneration` — create preview thumbnails
- `videoConversion` — transcode videos for streaming
- `ocr` — read text in photos
- `duplicateDetection` — find duplicate assets
- `sidecar` — process sidecar files (XMP, etc.)
- `library` — scan library for new/changed files
- `backupDatabase` — create DB backup
- `notifications` — process notification queue
- `storageTemplateMigration` — move files to organized folder structure

## GPU Verification

### Check ONNX Runtime CUDA availability inside container

```bash
docker exec immich_machine_learning bash -c \
  'source /opt/venv/bin/activate && \
   python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"'
```

Expected output for GPU-enabled:
```
GPU
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']
```

### Check GPU device files in container

```bash
docker exec immich_machine_learning ls -la /dev | grep nvidia
```

Should show: `nvidia0`, `nvidiactl`, `nvidia-uvm`, `nvidia-uvm-tools`

### Verify compose GPU config is attached

```bash
docker inspect immich_machine_learning --format '{{json .HostConfig.DeviceRequests}}'
```

Should show: `[{"Driver":"nvidia","Count":-1,...}]` for `count: all`

Check runtime: `docker inspect immich_machine_learning --format '{{json .HostConfig.Runtime}}'`

### Immich server container has Node.js, not Python

For inline JSON parsing inside `docker exec immich_server` commands, use Node:

```bash
# Instead of python3 -c (not available), use:
docker exec immich_server node -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$JSON_DATA"
```

## Docker Networking Recovery

If the Immich server cannot reach its database or redis (EHOSTUNREACH errors):
```
Error: connect EHOSTUNREACH 172.18.0.3:5432
Error: connect EHOSTUNREACH 172.18.0.4:6379
```

**Fix:** Restart the stack in dependency order:
```bash
cd /opt/immich && docker compose restart database redis
# Wait 5s for DB/redis to be ready
docker compose restart immich-server immich-machine-learning
```

## Docker Compose GPU Configuration (v2 format)

**Required** for the `v2-cuda` ML image to actually access the GPU:

```yaml
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
```

The `v2-cuda` image already has `DEVICE=cuda` and `NVIDIA_VISIBLE_DEVICES=all` in its Dockerfile ENV — no need to add those in compose. The missing piece is the `deploy.resources.reservations.devices` block.

**⚠️ Container has no `curl`:** The ML container has no `curl` or `ping` installed. For network tests, use Python's `urllib.request` from inside the activated venv:
```bash
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import urllib.request
r = urllib.request.urlopen('\''https://huggingface.co'\'', timeout=10)
print(r.status)
"'
```

## GPU Functional Verification

### Layered diagnostic procedure

When the ML container shows models failing to load, use this progression to isolate the issue:

**Layer 1 — GPU hardware is exposed to container:**
```bash
docker exec immich_machine_learning ls -la /dev | grep nvidia
# Must show: nvidia0, nvidiactl, nvidia-uvm
```

**Layer 2 — ONNX Runtime detects CUDA providers:**
```bash
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime
print(\"Device:\", onnxruntime.get_device())
print(\"Providers:\", onnxruntime.get_available_providers())
"'
# Expected: Device=GPU, Providers=[TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider]
```

**Layer 3 — Actual CUDA inference works:**
```bash
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime as ort, numpy as np, onnx
from onnx import helper, TensorProto

# Build a minimal model (Relu) and run it on CUDA
X = helper.make_tensor_value_info(\"X\", TensorProto.FLOAT, [1, 3])
Y = helper.make_tensor_value_info(\"Y\", TensorProto.FLOAT, [1, 3])
node = helper.make_node(\"Relu\", [\"X\"], [\"Y\"])
graph = helper.make_graph([node], \"test\", [X], [Y])
model = helper.make_model(graph)

sess = ort.InferenceSession(model.SerializeToString(),
  providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"])
result = sess.run(None, {\"X\": np.array([[-2.0, 0.0, 2.0]], dtype=np.float32)})
print(\"CUDA inference OK:\", result[0])
"'
# Expected: CUDA inference OK: [[0. 0. 2.]]
```

**Layer 4 — Load a real CLIP model on GPU (after download):**
```bash
# Download the CLIP model first (run this step if not yet cached)
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
from huggingface_hub import snapshot_download, os
cache_dir = \"/cache/clip/ViT-B-32__openai\"
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(\"immich-app/ViT-B-32__openai\", cache_dir=cache_dir, local_dir=cache_dir,
  ignore_patterns=[\"*.armnn\", \"*.rknn\"], max_workers=2)
"'

# Then load and run inference on GPU
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime as ort, numpy as np, time, os
model_path = \"/cache/clip/ViT-B-32__openai/visual/model.onnx\"
print(f\"Model: {os.path.getsize(model_path)/1024/1024:.0f} MB\")

so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
sess = ort.InferenceSession(model_path, sess_options=so,
  providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"],
  provider_options=[{\"device_id\": \"0\", \"gpu_mem_limit\": 3*1024*1024*1024}, {}])
print(f\"Loaded in {time.time():.0f}s\")

# Run one inference
data = np.random.randn(1, 3, 224, 224).astype(np.float32)
start = time.time()
outputs = sess.run(None, {sess.get_inputs()[0].name: data})
print(f\"Inference: {outputs[0].shape} in {(time.time()-start)*1000:.0f}ms\")
"'
# Expected: ~0.8s load, ~237ms inference, output shape (1, 512)
```

### Common failures at each layer

| Layer | Failure Symptom | Likely Cause |
|-------|----------------|--------------|
| 1 | No nvidia device files | GPU not configured in compose — add `deploy.resources.reservations.devices` |
| 1 | nvidia0 present but nvidia-smi not found | Container image has drivers but no nvidia-smi binary (normal for Immich ML image) |
| 2 | Only CPUExecutionProvider | GPU not exposed to container (wrong runtime / no device requests) |
| 3 | ONNX session creation fails | CUDA version mismatch (container CUDA 12.2 vs host driver) or OOM |
| 4 | Model download OK but load fails | Disk space, corrupt download, or model file mismatch. Check /cache mounts |
| 4 | Download loop (clear→retry→fail) | Multiple models writing to same cache_dir, overwriting each other's files. Try MACHINE_LEARNING_MODEL_ARENA=true