# Migrating from External AI APIs to llama-server

Pattern: replace external LLM API calls (Gemini, OpenAI, Anthropic) with a local llama-server behind nginx.

## When to use

- Broken/missing API keys in a legacy codebase
- Want to eliminate external API dependency for privacy, cost, or reliability
- Already have llama-server running locally (see `references/deployment-patterns.md`)

## Discovery: find all AI calls

```bash
# Find every fetch() or HTTP call in the project
grep -rn 'fetch(' --include='*.js' --include='*.html' .
# Focus on external AI APIs
grep -rn 'googleapis\|openai.com\|api.anthropic' --include='*.js' .
```

Real-world example from an auto-repair shop web app — 3 dead Gemini calls and 1 working local LLM call already in the codebase.

## Conversion recipe

### Gemini → llama.cpp

**Before (Gemini):**
```javascript
const apiKey = "AIzaSy...nZqc";
const apiUrl = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`;

const payload = {
    contents: [{
        role: "user",
        parts: [{ text: prompt }]
    }]
};

const response = await fetch(apiUrl, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(payload)
});

const result = await response.json();
const text = result.candidates[0].content.parts[0].text;
```

**After (llama-server):**
```javascript
const response = await fetch('/llm/v1/chat/completions', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
        messages: [
            {role:'system', content: prompt},       // instructions
            {role:'user', content: userData}        // actual data
        ],
        temperature: 0,
        max_tokens: 500
    })
});

const result = await response.json();
const text = result.choices[0].message.content;
```

### Response path mapping

| Provider | Response text path |
|---|---|
| Gemini | `result.candidates[0].content.parts[0].text` |
| OpenAI | `result.choices[0].message.content` |
| llama.cpp (`/v1/chat/completions`) | `result.choices[0].message.content` |

### Payload format mapping

| Provider | Format |
|---|---|
| Gemini | `{contents: [{role: "user", parts: [{text}]}]}` |
| OpenAI / llama.cpp | `{messages: [{role: "user", content: text}]}` |

## System prompt from user prompt

Gemini calls often have the full instruction text as a single `user`-role message. For llama.cpp, split it:

- **System message** = the instructions/formatting rules (the "you are an expert" part)
- **User message** = the actual data to process (service names, OCR text, etc.)

This mirrors the already-working pattern in the scan screenshot feature.

## Testing after migration

Always test end-to-end with the actual prompt the code uses:

```bash
# Test through nginx (what the browser sees)
curl -sk https://localhost:3447/llm/v1/chat/completions \
  -X POST -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"system","content":"Your prompt here"},{"role":"user","content":"Input data"}],"temperature":0,"max_tokens":300}'
```

Checklist:
- [ ] Response parses correctly (JSON keys match, LEVEL/EXPLANATION format intact)
- [ ] Error handling still works (network error, empty response)
- [ ] Existing fallbacks still trigger on failure

## Model quality expectations

1.5B models (Qwen2.5-1.5B) handle extraction and classification reasonably but:
- Priority/severity judgment is weaker than larger models
- May misclassify edge cases (2mm brake pads as "RECOMMENDED" not "CRITICAL")
- Solution: keep keyword-based fallback rules that override LLM judgments for known dangerous conditions

## Nginx prerequisite

The frontend uses a relative URL: `fetch('/llm/v1/chat/completions')`. This works because nginx serves both the static site and proxies `/llm/` to llama-server on the same port:

```nginx
location /llm/ {
    proxy_pass http://127.0.0.1:8081/;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_read_timeout 120s;
}
```

No CORS, no separate port, no API key management — the browser thinks it's calling its own server.