Migrating from External AI APIs to llama-server

Pattern: replace external LLM API calls (Gemini, OpenAI, Anthropic) with a local llama-server behind nginx.

When to use

Broken/missing API keys in a legacy codebase
Want to eliminate external API dependency for privacy, cost, or reliability
Already have llama-server running locally (see references/deployment-patterns.md)

Discovery: find all AI calls

# Find every fetch() or HTTP call in the project
grep -rn 'fetch(' --include='*.js' --include='*.html' .
# Focus on external AI APIs
grep -rn 'googleapis\|openai.com\|api.anthropic' --include='*.js' .

Real-world example from an auto-repair shop web app — 3 dead Gemini calls and 1 working local LLM call already in the codebase.

Conversion recipe

Gemini → llama.cpp

Before (Gemini):

const apiKey = "AIzaSy...nZqc";
const apiUrl = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`;

const payload = {
    contents: [{
        role: "user",
        parts: [{ text: prompt }]
    }]
};

const response = await fetch(apiUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload)
});

const result = await response.json();
const text = result.candidates[0].content.parts[0].text;

After (llama-server):

const response = await fetch('/llm/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        messages: [
            {role:'system', content: prompt},       // instructions
            {role:'user', content: userData}        // actual data
        ],
        temperature: 0,
        max_tokens: 500
    })
});

const result = await response.json();
const text = result.choices[0].message.content;

Response path mapping

Provider	Response text path
Gemini	`result.candidates[0].content.parts[0].text`
OpenAI	`result.choices[0].message.content`
llama.cpp (`/v1/chat/completions`)	`result.choices[0].message.content`

Payload format mapping

Provider	Format
Gemini	`{contents: [{role: "user", parts: [{text}]}]}`
OpenAI / llama.cpp	`{messages: [{role: "user", content: text}]}`

System prompt from user prompt

Gemini calls often have the full instruction text as a single user-role message. For llama.cpp, split it:

System message = the instructions/formatting rules (the "you are an expert" part)
User message = the actual data to process (service names, OCR text, etc.)

This mirrors the already-working pattern in the scan screenshot feature.

Testing after migration

Always test end-to-end with the actual prompt the code uses:

# Test through nginx (what the browser sees)
curl -sk https://localhost:3447/llm/v1/chat/completions \
  -X POST -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"system","content":"Your prompt here"},{"role":"user","content":"Input data"}],"temperature":0,"max_tokens":300}'

Checklist:

Response parses correctly (JSON keys match, LEVEL/EXPLANATION format intact)
Error handling still works (network error, empty response)
Existing fallbacks still trigger on failure

Model quality expectations

1.5B models (Qwen2.5-1.5B) handle extraction and classification reasonably but:

Priority/severity judgment is weaker than larger models
May misclassify edge cases (2mm brake pads as "RECOMMENDED" not "CRITICAL")
Solution: keep keyword-based fallback rules that override LLM judgments for known dangerous conditions

Nginx prerequisite

The frontend uses a relative URL: fetch('/llm/v1/chat/completions'). This works because nginx serves both the static site and proxies /llm/ to llama-server on the same port:

location /llm/ {
    proxy_pass http://127.0.0.1:8081/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_read_timeout 120s;
}

No CORS, no separate port, no API key management — the browser thinks it's calling its own server.

4.0 KiB Raw Blame History