hermes-config/skills/mlops/inference/llama-cpp/references/deployment-patterns.md

# llama.cpp Deployment Patterns

## CPU-only build (CUDA toolkit mismatch workaround)

When nvcc and CUDA libraries are from different versions (e.g., nvcc 12.4, libcublas.so from CUDA 13.1), the CUDA build will fail with linker errors like `undefined reference to cublasGemmEx@libcublas.so.12`.

**Fix:** Build CPU-only. For models ≤3B parameters, CPU inference is fast enough (20-40 tok/s).

```bash
git clone --depth 1 --branch <tag> https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j4
```

## Systemd service

```ini
[Unit]
Description=llama.cpp server for <model>
After=network.target

[Service]
Type=simple
User=<user>
ExecStart=<path>/llama-server \
    -m <path>/model.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -t 4
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

Install:
```bash
sudo cp llama-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
```

## Nginx reverse proxy

For browser access to llama-server (avoids CORS, works across devices):

```nginx
location /llm/ {
    proxy_pass http://127.0.0.1:8081/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout 120s;
}
```

Test: `curl -sk https://your-domain:port/llm/health`

## Browser integration

```javascript
const resp = await fetch('/llm/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        messages: [
            {role:'system', content:'Your prompt'},
            {role:'user', content: userText}
        ],
        temperature: 0,
        max_tokens: 500
    })
});
const data = await resp.json();
let raw = data.choices[0].message.content;
// Strip markdown fences (small models often wrap JSON)
raw = raw.replace(/^```(?:json)?\s*\n?/i, '').replace(/\n?```\s*$/i, '');
const parsed = JSON.parse(raw);
```

## OCR → LLM pipeline pattern

1. Tesseract.js runs OCR in browser → raw text
2. Browser POSTs text to `/llm/v1/chat/completions`
3. llama-server processes → returns structured JSON
4. Client normalizes fields (strip phone dashes, uppercase VIN)
5. Fallback to rule-based parser if LLM unavailable

## Prompt engineering for structured extraction

Key rules for reliable JSON extraction from small models (1.5B-3B):
- Return ONLY JSON, no markdown — but strip fences anyway
- Explicit field rules in system prompt (not just schema)
- Include example values in field descriptions
- Use temperature=0 for deterministic output
- `max_tokens=500` sufficient for 3-5 appointment records

Example system prompt for OCR appointment parsing:
```
Extract appointment details from OCR text. Return ONLY valid JSON, no markdown.

RULES:
- customerName: person name before phone number. Strip "RO" prefix.
- customerPhone: 10 digits only, no dashes
- customerEmail: actual email with @
- vin: 17-char uppercase VIN. OCR may misread 0 as O, 1 as I.
- vehicleInfo: year + make + model
- serviceType: work description after opcode
- opCode: only bracketed code like [REP] — no advisor names
- appointmentTime: 24h format (e.g. "09:00")
- duration: integer minutes

JSON: {"appointments":[{"customerName":"","customerPhone":"","customerEmail":"","vin":"","vehicleInfo":"","serviceType":"","opCode":"","appointmentTime":"","duration":0}]}
```

## Socket-activated llama-server (VRAM on-demand)

When the GPU is shared with other workloads (gaming, other models), keeping the model in VRAM 24/7 wastes resources. systemd socket activation starts llama-server only when a request hits the port, and auto-stops it after idle timeout.

**Cold-start latency:** 7-8s for a ~9 GB GGUF on PCIe 3.0 + NVMe (NVMe read ~3.5s, PCIe transfer ~0.7s, CUDA init ~3s). Warm restart (file cached in RAM): 3-4s. Acceptable for batch OCR and non-interactive use.

### Socket file

```ini
# /etc/systemd/system/llama-server.socket
[Socket]
ListenStream=127.0.0.1:8081

[Install]
WantedBy=sockets.target
```

### Service file (modified)

```ini
[Unit]
Description=llama.cpp server (socket-activated)
After=network.target

[Service]
Type=simple
User=ray
WorkingDirectory=/home/ray
ExecStart=/path/to/llama-server \
    -m /path/to/model.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -ngl 99 \
    -t 8
Restart=on-failure
RestartSec=5
StopIdleSec=300

[Install]
WantedBy=multi-user.target
```

**`StopIdleSec=300`** — systemd kills the service after 5 minutes with no active connections. Adjust up/down as needed.

### Activation

```bash
sudo systemctl stop llama-server
sudo systemctl disable llama-server
sudo systemctl enable --now llama-server.socket
# Now llama-server only launches when :8081 receives a connection
```

**Verification:** `ss -tlnp | grep 8081` shows the socket in LISTEN state with systemd as the listener. First request triggers service start.

### VRAM lifecycle

| State | VRAM |
|---|---|
| Idle (no connections) | 0 MB (service not running) |
| Active (handling requests) | Model size (~3-9 GB depending on quant) |
| Post-idle (StopIdleSec elapsed) | 0 MB (service killed, VRAM freed) |

### When NOT to socket-activate

- Interactive chat where 7-8s cold start is annoying
- Frequent bursts of requests (model reloads repeatedly)
- When the GPU has enough VRAM to leave the model resident permanently (e.g., 24 GB card with a 7B model)

## Multi-model deployment

Running two llama-server instances on different ports for different purposes (e.g., small model for chat + larger model for delegation/coding).

```bash
# Instance 1: main model (port 8081)
llama-server -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -c 4096 --port 8081 -ngl 99 -t 8 &

# Instance 2: delegation/coding model (port 8082)
llama-server -m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf -c 16384 --port 8082 -ngl 99 -t 8 &
```

### VRAM budgeting for dual models

Must fit within total GPU VRAM. Example on RTX 2080 Ti (11 GB):

| Combination | VRAM used | Fits? |
|---|---|---|
| 7B Q2_K (3.2 GB) + 7B Q4_K_M (4.7 GB) | ~7.9 GB | ✓ |
| 7B Q2_K (3.2 GB) + 14B Q4_K_M (9 GB) | ~12.2 GB | ✗ |
| 3B Q4_K_M (2 GB) + 14B Q4_K_M (9 GB) | ~11 GB | Tight |

For dual-model on a single GPU, prefer socket activation on one instance so they don't both stay loaded. Or accept that only one runs at a time.

### Hermes delegation config

Point Hermes subagents at a local delegation model:

```yaml
# ~/.hermes/config.yaml
delegation:
  provider: custom:local-delegation
  model: qwen-coder-14b
  base_url: http://127.0.0.1:8082/v1
  api_key: not-needed
  max_concurrent_children: 1  # local model can't parallelize well
```

Any OpenAI-compatible client can be pointed at llama-server the same way — set `base_url` to `http://127.0.0.1:<port>/v1` and use a placeholder API key.

### Quality expectations for delegation

A local 14B Q4 model vs a cloud API (e.g., DeepSeek V4 Flash):

| Dimension | Local 14B Q4 | Cloud API |
|---|---|---|
| Structured tasks (file ops, patches, grep) | Good | Excellent |
| Simple debugging | Good | Excellent |
| Complex multi-step reasoning | Fair | Excellent |
| Code generation (new features) | Good | Very good |
| Cost per delegation | $0 | API tokens |
| Latency | 45-55 tok/s local | API round-trip |

For delegation workloads (which are mostly structured), a 14B at Q4 is competent. The quality gap is real but often acceptable for the cost savings.

- GPU: GTX 1050 Ti 4GB VRAM
- RAM: 14GB
- Qwen2.5-1.5B-Instruct Q4_K_M: 941 MB, ~30 tok/s CPU, loads in ~500ms
- Llama-3.2-3B-Instruct Q4_K_M: ~2 GB, would fit in VRAM if CUDA worked