Ollama Deployment (Alternative to standalone llama-server)

Ollama provides on-demand model loading with automatic 5-minute idle unload. Unlike standalone llama-server which keeps a model resident in VRAM permanently, Ollama loads on first request and frees VRAM when idle — ideal for shared GPU scenarios (work + inference on the same card).

Installation

curl -fsSL https://ollama.com/install.sh | sh

Creates systemd service ollama, listens on 127.0.0.1:11434.

Pull a model

# Text inference
ollama pull qwen2.5:7b

# Vision (image understanding)
ollama pull llava:7b

Models auto-unload after 5 minutes idle. ollama ps shows currently loaded models.

Nginx reverse proxy — ⚠️ CRITICAL: Host header

Ollama rejects requests where the Host header is not localhost. When proxying through nginx, you MUST override the Host header:

location /llm/ {
    proxy_pass http://127.0.0.1:11434/;
    proxy_http_version 1.1;
    proxy_set_header Host "localhost";    # ← REQUIRED — Ollama rejects $host
    proxy_set_header X-Real-IP $remote_addr;
    proxy_buffering off;
    proxy_read_timeout 120s;              # allow for cold-start model load
}

Without this, nginx forwards the original Host header (grajmedia.duckdns.org, 192.168.50.98, etc.) and Ollama returns 403 Forbidden immediately (microsecond reject — not a model-loading delay).

Same fix applies for vision proxy:

location /vision/ {
    proxy_pass http://127.0.0.1:11434/;
    proxy_http_version 1.1;
    proxy_set_header Host "localhost";
    proxy_set_header X-Real-IP $remote_addr;
    proxy_buffering off;
    proxy_read_timeout 300s;              # vision models load slower
}

OpenAI-compatible endpoint

Ollama serves /v1/chat/completions with the same response format as llama-server and the OpenAI API:

{
  "id": "chatcmpl-898",
  "object": "chat.completion",
  "created": 1781059955,
  "model": "qwen2.5:7b",
  "choices": [{
    "index": 0,
    "message": { "role": "assistant", "content": "Hi!" },
    "finish_reason": "stop"
  }],
  "usage": { "prompt_tokens": 34, "completion_tokens": 3, "total_tokens": 37 }
}

Migrating from llama-server to Ollama

llama-server does not require a model field in the request body. Ollama does. When migrating, add the model name to every API call:

Before (llama-server):

body: JSON.stringify({
    messages: [...],
    temperature: 0,
    max_tokens: 500
})

After (Ollama):

body: JSON.stringify({
    model: 'qwen2.5:7b',
    messages: [...],
    temperature: 0,
    max_tokens: 500
})

Then update the nginx proxy target and restart:

# Change proxy_pass in nginx config, then:
sudo nginx -t && sudo systemctl reload nginx

Stop the old llama-server to free VRAM:

kill <llama-server-pid>

Vision model API

Ollama's vision API uses the native chat format with an images array. Base64 prefix data:image/png;base64, is NOT included — just the raw base64.

const b64 = await new Promise(resolve => {
    const r = new FileReader();
    r.onload = () => resolve(r.result.split(',')[1]);  // strip data:image/...;base64,
    r.readAsDataURL(file);
});

const resp = await fetch('/vision/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        model: 'llava:7b',
        messages: [{
            role: 'user',
            content: 'Your system prompt + instructions + schema',
            images: [b64]
        }],
        stream: false
    })
});
const data = await resp.json();
const content = data.message.content;  // note: not choices[0].message (that's /v1/chat/completions format)

Key difference: The native /api/chat endpoint returns data.message.content. The OpenAI-compatible /v1/chat/completions returns data.choices[0].message.content. Use /v1/chat/completions for text models and /api/chat for vision if you use the native format — but send the model field either way.

client_max_body_size

Vision requests include a base64 image which can be 2-5 MB (or more for large screenshots). nginx default client_max_body_size is 1 MB — requests over this are silently rejected with 413. Set in the server block:

client_max_body_size 50m;

Replacing Tesseract OCR with a vision model

For complex table layouts (schedules, invoices, spreadsheets), Tesseract cannot reliably preserve row/column structure. A VLM (vision-language model) like llava:7b sees the layout visually and outputs structured JSON directly, eliminating the need for:

Image preprocessing (upscaling, sharpening, contrast)
TSV bounding-box parsing
Coordinate-based row clustering
OCR text → LLM text → JSON pipeline

Old pipeline: Screenshot → preprocessing → Tesseract → TSV bbox clustering → text LLM → JSON

New pipeline: Screenshot → base64 → VLM → JSON

4.9 KiB Raw Blame History