ray
177 lines
4.9 KiB
Markdown
177 lines
4.9 KiB
Markdown
# Ollama Deployment (Alternative to standalone llama-server)
|
|
|
|
Ollama provides on-demand model loading with automatic 5-minute idle unload.
|
|
Unlike standalone llama-server which keeps a model resident in VRAM permanently,
|
|
Ollama loads on first request and frees VRAM when idle — ideal for shared GPU
|
|
scenarios (work + inference on the same card).
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
curl -fsSL https://ollama.com/install.sh | sh
|
|
```
|
|
|
|
Creates systemd service `ollama`, listens on `127.0.0.1:11434`.
|
|
|
|
## Pull a model
|
|
|
|
```bash
|
|
# Text inference
|
|
ollama pull qwen2.5:7b
|
|
|
|
# Vision (image understanding)
|
|
ollama pull llava:7b
|
|
```
|
|
|
|
Models auto-unload after 5 minutes idle. `ollama ps` shows currently loaded models.
|
|
|
|
## Nginx reverse proxy — ⚠️ CRITICAL: Host header
|
|
|
|
**Ollama rejects requests where the `Host` header is not `localhost`.** When
|
|
proxying through nginx, you MUST override the Host header:
|
|
|
|
```nginx
|
|
location /llm/ {
|
|
proxy_pass http://127.0.0.1:11434/;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Host "localhost"; # ← REQUIRED — Ollama rejects $host
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_buffering off;
|
|
proxy_read_timeout 120s; # allow for cold-start model load
|
|
}
|
|
```
|
|
|
|
Without this, nginx forwards the original Host header (`grajmedia.duckdns.org`,
|
|
`192.168.50.98`, etc.) and Ollama returns **403 Forbidden** immediately
|
|
(microsecond reject — not a model-loading delay).
|
|
|
|
Same fix applies for vision proxy:
|
|
|
|
```nginx
|
|
location /vision/ {
|
|
proxy_pass http://127.0.0.1:11434/;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Host "localhost";
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_buffering off;
|
|
proxy_read_timeout 300s; # vision models load slower
|
|
}
|
|
```
|
|
|
|
## OpenAI-compatible endpoint
|
|
|
|
Ollama serves `/v1/chat/completions` with the same response format as
|
|
llama-server and the OpenAI API:
|
|
|
|
```json
|
|
{
|
|
"id": "chatcmpl-898",
|
|
"object": "chat.completion",
|
|
"created": 1781059955,
|
|
"model": "qwen2.5:7b",
|
|
"choices": [{
|
|
"index": 0,
|
|
"message": { "role": "assistant", "content": "Hi!" },
|
|
"finish_reason": "stop"
|
|
}],
|
|
"usage": { "prompt_tokens": 34, "completion_tokens": 3, "total_tokens": 37 }
|
|
}
|
|
```
|
|
|
|
## Migrating from llama-server to Ollama
|
|
|
|
llama-server does not require a `model` field in the request body. Ollama does.
|
|
When migrating, add the model name to every API call:
|
|
|
|
**Before (llama-server):**
|
|
```javascript
|
|
body: JSON.stringify({
|
|
messages: [...],
|
|
temperature: 0,
|
|
max_tokens: 500
|
|
})
|
|
```
|
|
|
|
**After (Ollama):**
|
|
```javascript
|
|
body: JSON.stringify({
|
|
model: 'qwen2.5:7b',
|
|
messages: [...],
|
|
temperature: 0,
|
|
max_tokens: 500
|
|
})
|
|
```
|
|
|
|
Then update the nginx proxy target and restart:
|
|
|
|
```bash
|
|
# Change proxy_pass in nginx config, then:
|
|
sudo nginx -t && sudo systemctl reload nginx
|
|
```
|
|
|
|
Stop the old llama-server to free VRAM:
|
|
|
|
```bash
|
|
kill <llama-server-pid>
|
|
```
|
|
|
|
## Vision model API
|
|
|
|
Ollama's vision API uses the native chat format with an `images` array.
|
|
Base64 prefix `data:image/png;base64,` is NOT included — just the raw base64.
|
|
|
|
```javascript
|
|
const b64 = await new Promise(resolve => {
|
|
const r = new FileReader();
|
|
r.onload = () => resolve(r.result.split(',')[1]); // strip data:image/...;base64,
|
|
r.readAsDataURL(file);
|
|
});
|
|
|
|
const resp = await fetch('/vision/api/chat', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({
|
|
model: 'llava:7b',
|
|
messages: [{
|
|
role: 'user',
|
|
content: 'Your system prompt + instructions + schema',
|
|
images: [b64]
|
|
}],
|
|
stream: false
|
|
})
|
|
});
|
|
const data = await resp.json();
|
|
const content = data.message.content; // note: not choices[0].message (that's /v1/chat/completions format)
|
|
```
|
|
|
|
**Key difference:** The native `/api/chat` endpoint returns `data.message.content`.
|
|
The OpenAI-compatible `/v1/chat/completions` returns `data.choices[0].message.content`.
|
|
Use `/v1/chat/completions` for text models and `/api/chat` for vision if you use
|
|
the native format — but send the `model` field either way.
|
|
|
|
## client_max_body_size
|
|
|
|
Vision requests include a base64 image which can be 2-5 MB (or more for large
|
|
screenshots). nginx default `client_max_body_size` is 1 MB — requests over this
|
|
are silently rejected with 413. Set in the server block:
|
|
|
|
```nginx
|
|
client_max_body_size 50m;
|
|
```
|
|
|
|
## Replacing Tesseract OCR with a vision model
|
|
|
|
For complex table layouts (schedules, invoices, spreadsheets), Tesseract cannot
|
|
reliably preserve row/column structure. A VLM (vision-language model) like
|
|
llava:7b sees the layout visually and outputs structured JSON directly,
|
|
eliminating the need for:
|
|
- Image preprocessing (upscaling, sharpening, contrast)
|
|
- TSV bounding-box parsing
|
|
- Coordinate-based row clustering
|
|
- OCR text → LLM text → JSON pipeline
|
|
|
|
**Old pipeline:** Screenshot → preprocessing → Tesseract → TSV bbox clustering →
|
|
text LLM → JSON
|
|
|
|
**New pipeline:** Screenshot → base64 → VLM → JSON
|