hermes-config/skills/mlops/inference/llama-cpp/references/deployment-patterns.md

llama.cpp Deployment Patterns

CPU-only build (CUDA toolkit mismatch workaround)

When nvcc and CUDA libraries are from different versions (e.g., nvcc 12.4, libcublas.so from CUDA 13.1), the CUDA build will fail with linker errors like undefined reference to cublasGemmEx@libcublas.so.12.

Fix: Build CPU-only. For models ≤3B parameters, CPU inference is fast enough (20-40 tok/s).

git clone --depth 1 --branch <tag> https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j4

Systemd service

[Unit]
Description=llama.cpp server for <model>
After=network.target

[Service]
Type=simple
User=<user>
ExecStart=<path>/llama-server \
    -m <path>/model.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -t 4
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Install:

sudo cp llama-server.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server

Nginx reverse proxy

For browser access to llama-server (avoids CORS, works across devices):

location /llm/ {
    proxy_pass http://127.0.0.1:8081/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_read_timeout 120s;
}

Test: curl -sk https://your-domain:port/llm/health

Browser integration

const resp = await fetch('/llm/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        messages: [
            {role:'system', content:'Your prompt'},
            {role:'user', content: userText}
        ],
        temperature: 0,
        max_tokens: 500
    })
});
const data = await resp.json();
let raw = data.choices[0].message.content;
// Strip markdown fences (small models often wrap JSON)
raw = raw.replace(/^```(?:json)?\s*\n?/i, '').replace(/\n?```\s*$/i, '');
const parsed = JSON.parse(raw);

OCR → LLM pipeline pattern

1. Tesseract.js runs OCR in browser → raw text
2. Browser POSTs text to /llm/v1/chat/completions
3. llama-server processes → returns structured JSON
4. Client normalizes fields (strip phone dashes, uppercase VIN)
5. Fallback to rule-based parser if LLM unavailable

Prompt engineering for structured extraction

Key rules for reliable JSON extraction from small models (1.5B-3B):

• Return ONLY JSON, no markdown — but strip fences anyway
• Explicit field rules in system prompt (not just schema)
• Include example values in field descriptions
• Use temperature=0 for deterministic output
• max_tokens=500 sufficient for 3-5 appointment records

Example system prompt for OCR appointment parsing:

Extract appointment details from OCR text. Return ONLY valid JSON, no markdown.

RULES:
- customerName: person name before phone number. Strip "RO" prefix.
- customerPhone: 10 digits only, no dashes
- customerEmail: actual email with @
- vin: 17-char uppercase VIN. OCR may misread 0 as O, 1 as I.
- vehicleInfo: year + make + model
- serviceType: work description after opcode
- opCode: only bracketed code like [REP] — no advisor names
- appointmentTime: 24h format (e.g. "09:00")
- duration: integer minutes

JSON: {"appointments":[{"customerName":"","customerPhone":"","customerEmail":"","vin":"","vehicleInfo":"","serviceType":"","opCode":"","appointmentTime":"","duration":0}]}

Socket-activated llama-server (VRAM on-demand)

When the GPU is shared with other workloads (gaming, other models), keeping the model in VRAM 24/7 wastes resources. systemd socket activation starts llama-server only when a request hits the port, and auto-stops it after idle timeout.

Cold-start latency: 7-8s for a ~9 GB GGUF on PCIe 3.0 + NVMe (NVMe read ~3.5s, PCIe transfer ~0.7s, CUDA init ~3s). Warm restart (file cached in RAM): 3-4s. Acceptable for batch OCR and non-interactive use.

Socket file

# /etc/systemd/system/llama-server.socket
[Socket]
ListenStream=127.0.0.1:8081

[Install]
WantedBy=sockets.target

Service file (modified)

[Unit]
Description=llama.cpp server (socket-activated)
After=network.target

[Service]
Type=simple
User=ray
WorkingDirectory=/home/ray
ExecStart=/path/to/llama-server \
    -m /path/to/model.gguf \
    -c 4096 \
    --host 127.0.0.1 \
    --port 8081 \
    -ngl 99 \
    -t 8
Restart=on-failure
RestartSec=5
StopIdleSec=300

[Install]
WantedBy=multi-user.target

StopIdleSec=300 — systemd kills the service after 5 minutes with no active connections. Adjust up/down as needed.

Activation

sudo systemctl stop llama-server
sudo systemctl disable llama-server
sudo systemctl enable --now llama-server.socket
# Now llama-server only launches when :8081 receives a connection

Verification: ss -tlnp | grep 8081 shows the socket in LISTEN state with systemd as the listener. First request triggers service start.

VRAM lifecycle

State	VRAM
Idle (no connections)	0 MB (service not running)
Active (handling requests)	Model size (~3-9 GB depending on quant)
Post-idle (StopIdleSec elapsed)	0 MB (service killed, VRAM freed)

When NOT to socket-activate

• Interactive chat where 7-8s cold start is annoying
• Frequent bursts of requests (model reloads repeatedly)
• When the GPU has enough VRAM to leave the model resident permanently (e.g., 24 GB card with a 7B model)

Multi-model deployment

Running two llama-server instances on different ports for different purposes (e.g., small model for chat + larger model for delegation/coding).

# Instance 1: main model (port 8081)
llama-server -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -c 4096 --port 8081 -ngl 99 -t 8 &

# Instance 2: delegation/coding model (port 8082)
llama-server -m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf -c 16384 --port 8082 -ngl 99 -t 8 &

VRAM budgeting for dual models

Must fit within total GPU VRAM. Example on RTX 2080 Ti (11 GB):

Combination	VRAM used	Fits?
7B Q2_K (3.2 GB) + 7B Q4_K_M (4.7 GB)	~7.9 GB	✓
7B Q2_K (3.2 GB) + 14B Q4_K_M (9 GB)	~12.2 GB	✗
3B Q4_K_M (2 GB) + 14B Q4_K_M (9 GB)	~11 GB	Tight

For dual-model on a single GPU, prefer socket activation on one instance so they don't both stay loaded. Or accept that only one runs at a time.

Hermes delegation config

Point Hermes subagents at a local delegation model:

# ~/.hermes/config.yaml
delegation:
  provider: custom:local-delegation
  model: qwen-coder-14b
  base_url: http://127.0.0.1:8082/v1
  api_key: not-needed
  max_concurrent_children: 1  # local model can't parallelize well

Any OpenAI-compatible client can be pointed at llama-server the same way — set base_url to http://127.0.0.1:<port>/v1 and use a placeholder API key.

Quality expectations for delegation

A local 14B Q4 model vs a cloud API (e.g., DeepSeek V4 Flash):

Dimension	Local 14B Q4	Cloud API
Structured tasks (file ops, patches, grep)	Good	Excellent
Simple debugging	Good	Excellent
Complex multi-step reasoning	Fair	Excellent
Code generation (new features)	Good	Very good
Cost per delegation	$0	API tokens
Latency	45-55 tok/s local	API round-trip

For delegation workloads (which are mostly structured), a 14B at Q4 is competent. The quality gap is real but often acceptable for the cost savings.

• GPU: GTX 1050 Ti 4GB VRAM
• RAM: 14GB
• Qwen2.5-1.5B-Instruct Q4_K_M: 941 MB, ~30 tok/s CPU, loads in ~500ms
• Llama-3.2-3B-Instruct Q4_K_M: ~2 GB, would fit in VRAM if CUDA worked