hermes-config/skills/software-development/immich-server/references/immich-api-jobs.md

Immich API: Job Management & GPU Troubleshooting

Session-specific commands for triggering ML jobs and diagnosing GPU acceleration on the Immich machine-learning container (v2.7.5+).

Authentication

Login (get bearer token)

LOGIN=$(curl -s -X POST http://localhost:2283/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email":"admin@example.com","password":"your-password"}')
TOK=*** -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$LOGIN")

The token is a bearer token, used as `Authorization: Bearer *** all subsequent API calls.

Job Management

List all job queues

curl -s http://localhost:2283/api/jobs -H "Authorization: Bearer ***

Returns each job name with queueStatus (isPaused, isActive) and jobCounts (active, completed, failed, delayed, waiting, paused).

Trigger a specific job

curl -s -X PUT "http://localhost:2283/api/jobs/smartSearch" \
  -H "Authorization: Bearer *** \
  -H "Content-Type: application/json" \
  -d '{"command":"start"}'

To force re-process already-completed assets, use {"command":"start","force":true}.

Available job names

• smartSearch — CLIP embeddings for natural language search
• faceDetection — detect faces in photos
• facialRecognition — group detected faces into people
• metadataExtraction — read EXIF dates, GPS, camera info
• thumbnailGeneration — create preview thumbnails
• videoConversion — transcode videos for streaming
• ocr — read text in photos
• duplicateDetection — find duplicate assets
• sidecar — process sidecar files (XMP, etc.)
• library — scan library for new/changed files
• backupDatabase — create DB backup
• notifications — process notification queue
• storageTemplateMigration — move files to organized folder structure

GPU Verification

Check ONNX Runtime CUDA availability inside container

docker exec immich_machine_learning bash -c \
  'source /opt/venv/bin/activate && \
   python -c "import onnxruntime; print(onnxruntime.get_device()); print(onnxruntime.get_available_providers())"'

Expected output for GPU-enabled:

GPU
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']

Check GPU device files in container

docker exec immich_machine_learning ls -la /dev | grep nvidia

Should show: nvidia0, nvidiactl, nvidia-uvm, nvidia-uvm-tools

Verify compose GPU config is attached

docker inspect immich_machine_learning --format '{{json .HostConfig.DeviceRequests}}'

Should show: [{"Driver":"nvidia","Count":-1,...}] for count: all

Check runtime: docker inspect immich_machine_learning --format '{{json .HostConfig.Runtime}}'

Immich server container has Node.js, not Python

For inline JSON parsing inside docker exec immich_server commands, use Node:

# Instead of python3 -c (not available), use:
docker exec immich_server node -e "console.log(JSON.parse(process.argv[1]).accessToken)" -- "$JSON_DATA"

Docker Networking Recovery

If the Immich server cannot reach its database or redis (EHOSTUNREACH errors):

Error: connect EHOSTUNREACH 172.18.0.3:5432
Error: connect EHOSTUNREACH 172.18.0.4:6379

Fix: Restart the stack in dependency order:

cd /opt/immich && docker compose restart database redis
# Wait 5s for DB/redis to be ready
docker compose restart immich-server immich-machine-learning

Docker Compose GPU Configuration (v2 format)

Required for the v2-cuda ML image to actually access the GPU:

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

The v2-cuda image already has DEVICE=cuda and NVIDIA_VISIBLE_DEVICES=all in its Dockerfile ENV — no need to add those in compose. The missing piece is the deploy.resources.reservations.devices block.

⚠️ Container has no curl: The ML container has no curl or ping installed. For network tests, use Python's urllib.request from inside the activated venv:

docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import urllib.request
r = urllib.request.urlopen('\''https://huggingface.co'\'', timeout=10)
print(r.status)
"'

GPU Functional Verification

Layered diagnostic procedure

When the ML container shows models failing to load, use this progression to isolate the issue:

Layer 1 — GPU hardware is exposed to container:

docker exec immich_machine_learning ls -la /dev | grep nvidia
# Must show: nvidia0, nvidiactl, nvidia-uvm

Layer 2 — ONNX Runtime detects CUDA providers:

docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime
print(\"Device:\", onnxruntime.get_device())
print(\"Providers:\", onnxruntime.get_available_providers())
"'
# Expected: Device=GPU, Providers=[TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider]

Layer 3 — Actual CUDA inference works:

docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime as ort, numpy as np, onnx
from onnx import helper, TensorProto

# Build a minimal model (Relu) and run it on CUDA
X = helper.make_tensor_value_info(\"X\", TensorProto.FLOAT, [1, 3])
Y = helper.make_tensor_value_info(\"Y\", TensorProto.FLOAT, [1, 3])
node = helper.make_node(\"Relu\", [\"X\"], [\"Y\"])
graph = helper.make_graph([node], \"test\", [X], [Y])
model = helper.make_model(graph)

sess = ort.InferenceSession(model.SerializeToString(),
  providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"])
result = sess.run(None, {\"X\": np.array([[-2.0, 0.0, 2.0]], dtype=np.float32)})
print(\"CUDA inference OK:\", result[0])
"'
# Expected: CUDA inference OK: [[0. 0. 2.]]

Layer 4 — Load a real CLIP model on GPU (after download):

# Download the CLIP model first (run this step if not yet cached)
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
from huggingface_hub import snapshot_download, os
cache_dir = \"/cache/clip/ViT-B-32__openai\"
os.makedirs(cache_dir, exist_ok=True)
snapshot_download(\"immich-app/ViT-B-32__openai\", cache_dir=cache_dir, local_dir=cache_dir,
  ignore_patterns=[\"*.armnn\", \"*.rknn\"], max_workers=2)
"'

# Then load and run inference on GPU
docker exec immich_machine_learning bash -c 'source /opt/venv/bin/activate && python -c "
import onnxruntime as ort, numpy as np, time, os
model_path = \"/cache/clip/ViT-B-32__openai/visual/model.onnx\"
print(f\"Model: {os.path.getsize(model_path)/1024/1024:.0f} MB\")

so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
sess = ort.InferenceSession(model_path, sess_options=so,
  providers=[\"CUDAExecutionProvider\", \"CPUExecutionProvider\"],
  provider_options=[{\"device_id\": \"0\", \"gpu_mem_limit\": 3*1024*1024*1024}, {}])
print(f\"Loaded in {time.time():.0f}s\")

# Run one inference
data = np.random.randn(1, 3, 224, 224).astype(np.float32)
start = time.time()
outputs = sess.run(None, {sess.get_inputs()[0].name: data})
print(f\"Inference: {outputs[0].shape} in {(time.time()-start)*1000:.0f}ms\")
"'
# Expected: ~0.8s load, ~237ms inference, output shape (1, 512)

Common failures at each layer

Layer	Failure Symptom	Likely Cause
1	No nvidia device files	GPU not configured in compose — add deploy.resources.reservations.devices
1	nvidia0 present but nvidia-smi not found	Container image has drivers but no nvidia-smi binary (normal for Immich ML image)
2	Only CPUExecutionProvider	GPU not exposed to container (wrong runtime / no device requests)
3	ONNX session creation fails	CUDA version mismatch (container CUDA 12.2 vs host driver) or OOM
4	Model download OK but load fails	Disk space, corrupt download, or model file mismatch. Check /cache mounts
4	Download loop (clear→retry→fail)	Multiple models writing to same cache_dir, overwriting each other's files. Try MACHINE_LEARNING_MODEL_ARENA=true