hermes-config/skills/software-development/immich-server/references/postgres-diagnostics.md

# Immich Postgres Diagnostics (No API Key)

When you don't have an Immich API key handy, you can check server health, job progress,
ML state, and config directly via the Postgres database inside the Immich Postgres container.

## Finding DB Credentials

The DB creds are set on the Postgres container at creation time. Get them from Docker:

```bash
docker inspect immich_postgres --format '{{json .Config.Env}}' | python3 -c "
import json,sys
env = json.load(sys.stdin)
for e in env:
    if 'POSTGRES' in e.upper() or 'DB_' in e:
        print(e)
"
```

Expected output:
```
POSTGRES_PASSWORD=ab496a...17a2
POSTGRES_USER=postgres
POSTGRES_DB=immich
```

Run queries with:
```bash
docker exec immich_postgres psql -U postgres -d immich -c "SQL HERE"
```

If password is required, pass via env:
```bash
docker exec -e PGPASSWORD=PASS immich_postgres psql -U postgres -d immich -c "SQL HERE"
```

## Job Queue Progress (per-asset job status)

Immich v2.7+ tracks per-asset job completion in `asset_job_status`. This is NOT a
traditional queue — it shows which assets have had each job type complete:

```sql
SELECT 
  COUNT(*) as total_assets,
  COUNT(*) FILTER (WHERE ajs."facesRecognizedAt" IS NOT NULL) as faces_done,
  COUNT(*) FILTER (WHERE ajs."metadataExtractedAt" IS NOT NULL) as metadata_done,
  COUNT(*) FILTER (WHERE ajs."duplicatesDetectedAt" IS NOT NULL) as dedup_done,
  COUNT(*) FILTER (WHERE ajs."ocrAt" IS NOT NULL) as ocr_done
FROM asset a
LEFT JOIN asset_job_status ajs ON a.id = ajs."assetId";
```

**Table schema** (`\d asset_job_status`):
```
assetId              | uuid                    | not null
facesRecognizedAt    | timestamp with time zone|
metadataExtractedAt  | timestamp with time zone|
duplicatesDetectedAt | timestamp with time zone|
ocrAt                | timestamp with time zone|
```

## System Metadata

The `system_metadata` table holds Immich's key state as JSONB:

```sql
SELECT * FROM system_metadata;
```

Key entries:

| key | value (snippet) | purpose |
|-----|-----------------|---------|
| `system-config` | `{"ffmpeg": {"transcode":"disabled"}, "storageTemplate": {"enabled": true}}` | Server config |
| `facial-recognition-state` | `{"lastRun": "2026-05-31T00:00:05.175Z"}` | Last ML face run |
| `reverse-geocoding-state` | `{"lastUpdate": "...", "lastImportFileName": "cities500.txt"}` | Geo data freshness |
| `version-check-state` | `{"checkedAt": "...", "releaseVersion": "v2.7.5"}` | Version tracking |
| `memories-state` | `{"lastOnThisDayDate": "..."}` | Memories feature state |
| `system-flags` | `{"mountChecks": {"thumbs": true, "upload": true, ...}}` | Mount health |
| `admin-onboarding` | `{"isOnboarded": true}` | Admin setup status |

To extract just the ML config:
```sql
SELECT json_data -> 'machineLearning' as ml_config FROM system_metadata WHERE key='system-config';
```

## Asset & People Counts

```sql
SELECT COUNT(*) as total_assets FROM asset;
SELECT COUNT(*) as total_albums FROM album;
SELECT COUNT(*) as total_people FROM person;
```

## ML Container GPU Check

The ML container uses ONNX Runtime (not PyTorch directly). Check GPU availability:

```bash
docker exec immich_machine_learning python3 -c "
import onnxruntime as ort
print(f'ONNX Runtime: {ort.__version__}')
print(f'Providers: {ort.get_available_providers()}')
print(f'CUDA: {\"CUDAExecutionProvider\" in ort.get_available_providers()}')
"
```

Check the container's configured device mode:
```bash
docker inspect immich_machine_learning --format '{{json .Config.Env}}' | python3 -c "
import json,sys
env = json.load(sys.stdin)
for e in env:
    if 'DEVICE' in e or 'GPU' in e or 'CUDA' in e:
        print(e)
"
```

Expected output for GPU-enabled:
```
DEVICE=cuda
NVIDIA_VISIBLE_DEVICES=all
CUDA_VERSION=12.2.2
```

ONNX Runtime 1.24+ typically offers `TensorrtExecutionProvider`, `CUDAExecutionProvider`,
`CPUExecutionProvider`. If `CUDAExecutionProvider` is in the list, GPU ML is active.

## Quick All-in-One Health Check

```bash
docker exec immich_postgres psql -U postgres -d immich -c "
SELECT 'assets' as check, COUNT(*)::text FROM asset
UNION ALL
SELECT 'people', COUNT(*)::text FROM person
UNION ALL
SELECT 'albums', COUNT(*)::text FROM album
UNION ALL
SELECT 'faces_done', COUNT(*) FILTER (WHERE ajs.\"facesRecognizedAt\" IS NOT NULL)::text
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
UNION ALL
SELECT 'metadata_done', COUNT(*) FILTER (WHERE ajs.\"metadataExtractedAt\" IS NOT NULL)::text
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
UNION ALL
SELECT 'ocr_done', COUNT(*) FILTER (WHERE ajs.\"ocrAt\" IS NOT NULL)::text
FROM asset a LEFT JOIN asset_job_status ajs ON a.id = ajs.\"assetId\"
UNION ALL
SELECT 'face_last_run', value->>'lastRun' FROM system_metadata WHERE key='facial-recognition-state'
UNION ALL
SELECT 'version', value->>'releaseVersion' FROM system_metadata WHERE key='version-check-state'
UNION ALL
SELECT 'transcoding', CASE WHEN value->'ffmpeg'->>'transcode' = 'disabled' THEN 'disabled' ELSE 'enabled' END
FROM system_metadata WHERE key='system-config';
"
```

## When to use Postgres diagnostics vs API

| Situation | Best approach |
|-----------|---------------|
| You have an API key | `/api/jobs` endpoint (richer data: active, waiting, failed, paused counts) |
| No API key / lost access | Postgres queries above |
| Checking ML container health | `docker exec` + ONNX Runtime check |
| GPU detection | `docker inspect` ML container env vars |