initial commit

skills/mlops/inference/llama-cpp/references/ollama-deployment.md

@@ -0,0 +1,176 @@
+# Ollama Deployment (Alternative to standalone llama-server)
+
+Ollama provides on-demand model loading with automatic 5-minute idle unload.
+Unlike standalone llama-server which keeps a model resident in VRAM permanently,
+Ollama loads on first request and frees VRAM when idle — ideal for shared GPU
+scenarios (work + inference on the same card).
+
+## Installation
+
+```bash
+curl -fsSL https://ollama.com/install.sh | sh
+```
+
+Creates systemd service `ollama`, listens on `127.0.0.1:11434`.
+
+## Pull a model
+
+```bash
+# Text inference
+ollama pull qwen2.5:7b
+
+# Vision (image understanding)
+ollama pull llava:7b
+```
+
+Models auto-unload after 5 minutes idle. `ollama ps` shows currently loaded models.
+
+## Nginx reverse proxy — ⚠️ CRITICAL: Host header
+
+**Ollama rejects requests where the `Host` header is not `localhost`.** When
+proxying through nginx, you MUST override the Host header:
+
+```nginx
+location /llm/ {
+    proxy_pass http://127.0.0.1:11434/;
+    proxy_http_version 1.1;
+    proxy_set_header Host "localhost";    # ← REQUIRED — Ollama rejects $host
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_buffering off;
+    proxy_read_timeout 120s;              # allow for cold-start model load
+}
+```
+
+Without this, nginx forwards the original Host header (`grajmedia.duckdns.org`,
+`192.168.50.98`, etc.) and Ollama returns **403 Forbidden** immediately
+(microsecond reject — not a model-loading delay).
+
+Same fix applies for vision proxy:
+
+```nginx
+location /vision/ {
+    proxy_pass http://127.0.0.1:11434/;
+    proxy_http_version 1.1;
+    proxy_set_header Host "localhost";
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_buffering off;
+    proxy_read_timeout 300s;              # vision models load slower
+}
+```
+
+## OpenAI-compatible endpoint
+
+Ollama serves `/v1/chat/completions` with the same response format as
+llama-server and the OpenAI API:
+
+```json
+{
+  "id": "chatcmpl-898",
+  "object": "chat.completion",
+  "created": 1781059955,
+  "model": "qwen2.5:7b",
+  "choices": [{
+    "index": 0,
+    "message": { "role": "assistant", "content": "Hi!" },
+    "finish_reason": "stop"
+  }],
+  "usage": { "prompt_tokens": 34, "completion_tokens": 3, "total_tokens": 37 }
+}
+```
+
+## Migrating from llama-server to Ollama
+
+llama-server does not require a `model` field in the request body. Ollama does.
+When migrating, add the model name to every API call:
+
+**Before (llama-server):**
+```javascript
+body: JSON.stringify({
+    messages: [...],
+    temperature: 0,
+    max_tokens: 500
+})
+```
+
+**After (Ollama):**
+```javascript
+body: JSON.stringify({
+    model: 'qwen2.5:7b',
+    messages: [...],
+    temperature: 0,
+    max_tokens: 500
+})
+```
+
+Then update the nginx proxy target and restart:
+
+```bash
+# Change proxy_pass in nginx config, then:
+sudo nginx -t && sudo systemctl reload nginx
+```
+
+Stop the old llama-server to free VRAM:
+
+```bash
+kill <llama-server-pid>
+```
+
+## Vision model API
+
+Ollama's vision API uses the native chat format with an `images` array.
+Base64 prefix `data:image/png;base64,` is NOT included — just the raw base64.
+
+```javascript
+const b64 = await new Promise(resolve => {
+    const r = new FileReader();
+    r.onload = () => resolve(r.result.split(',')[1]);  // strip data:image/...;base64,
+    r.readAsDataURL(file);
+});
+
+const resp = await fetch('/vision/api/chat', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+        model: 'llava:7b',
+        messages: [{
+            role: 'user',
+            content: 'Your system prompt + instructions + schema',
+            images: [b64]
+        }],
+        stream: false
+    })
+});
+const data = await resp.json();
+const content = data.message.content;  // note: not choices[0].message (that's /v1/chat/completions format)
+```
+
+**Key difference:** The native `/api/chat` endpoint returns `data.message.content`.
+The OpenAI-compatible `/v1/chat/completions` returns `data.choices[0].message.content`.
+Use `/v1/chat/completions` for text models and `/api/chat` for vision if you use
+the native format — but send the `model` field either way.
+
+## client_max_body_size
+
+Vision requests include a base64 image which can be 2-5 MB (or more for large
+screenshots). nginx default `client_max_body_size` is 1 MB — requests over this
+are silently rejected with 413. Set in the server block:
+
+```nginx
+client_max_body_size 50m;
+```
+
+## Replacing Tesseract OCR with a vision model
+
+For complex table layouts (schedules, invoices, spreadsheets), Tesseract cannot
+reliably preserve row/column structure. A VLM (vision-language model) like
+llava:7b sees the layout visually and outputs structured JSON directly,
+eliminating the need for:
+- Image preprocessing (upscaling, sharpening, contrast)
+- TSV bounding-box parsing
+- Coordinate-based row clustering
+- OCR text → LLM text → JSON pipeline
+
+**Old pipeline:** Screenshot → preprocessing → Tesseract → TSV bbox clustering →
+text LLM → JSON
+
+**New pipeline:** Screenshot → base64 → VLM → JSON