ray
128 lines
4.0 KiB
Markdown
128 lines
4.0 KiB
Markdown
# Migrating from External AI APIs to llama-server
|
|
|
|
Pattern: replace external LLM API calls (Gemini, OpenAI, Anthropic) with a local llama-server behind nginx.
|
|
|
|
## When to use
|
|
|
|
- Broken/missing API keys in a legacy codebase
|
|
- Want to eliminate external API dependency for privacy, cost, or reliability
|
|
- Already have llama-server running locally (see `references/deployment-patterns.md`)
|
|
|
|
## Discovery: find all AI calls
|
|
|
|
```bash
|
|
# Find every fetch() or HTTP call in the project
|
|
grep -rn 'fetch(' --include='*.js' --include='*.html' .
|
|
# Focus on external AI APIs
|
|
grep -rn 'googleapis\|openai.com\|api.anthropic' --include='*.js' .
|
|
```
|
|
|
|
Real-world example from an auto-repair shop web app — 3 dead Gemini calls and 1 working local LLM call already in the codebase.
|
|
|
|
## Conversion recipe
|
|
|
|
### Gemini → llama.cpp
|
|
|
|
**Before (Gemini):**
|
|
```javascript
|
|
const apiKey = "AIzaSy...nZqc";
|
|
const apiUrl = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`;
|
|
|
|
const payload = {
|
|
contents: [{
|
|
role: "user",
|
|
parts: [{ text: prompt }]
|
|
}]
|
|
};
|
|
|
|
const response = await fetch(apiUrl, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify(payload)
|
|
});
|
|
|
|
const result = await response.json();
|
|
const text = result.candidates[0].content.parts[0].text;
|
|
```
|
|
|
|
**After (llama-server):**
|
|
```javascript
|
|
const response = await fetch('/llm/v1/chat/completions', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({
|
|
messages: [
|
|
{role:'system', content: prompt}, // instructions
|
|
{role:'user', content: userData} // actual data
|
|
],
|
|
temperature: 0,
|
|
max_tokens: 500
|
|
})
|
|
});
|
|
|
|
const result = await response.json();
|
|
const text = result.choices[0].message.content;
|
|
```
|
|
|
|
### Response path mapping
|
|
|
|
| Provider | Response text path |
|
|
|---|---|
|
|
| Gemini | `result.candidates[0].content.parts[0].text` |
|
|
| OpenAI | `result.choices[0].message.content` |
|
|
| llama.cpp (`/v1/chat/completions`) | `result.choices[0].message.content` |
|
|
|
|
### Payload format mapping
|
|
|
|
| Provider | Format |
|
|
|---|---|
|
|
| Gemini | `{contents: [{role: "user", parts: [{text}]}]}` |
|
|
| OpenAI / llama.cpp | `{messages: [{role: "user", content: text}]}` |
|
|
|
|
## System prompt from user prompt
|
|
|
|
Gemini calls often have the full instruction text as a single `user`-role message. For llama.cpp, split it:
|
|
|
|
- **System message** = the instructions/formatting rules (the "you are an expert" part)
|
|
- **User message** = the actual data to process (service names, OCR text, etc.)
|
|
|
|
This mirrors the already-working pattern in the scan screenshot feature.
|
|
|
|
## Testing after migration
|
|
|
|
Always test end-to-end with the actual prompt the code uses:
|
|
|
|
```bash
|
|
# Test through nginx (what the browser sees)
|
|
curl -sk https://localhost:3447/llm/v1/chat/completions \
|
|
-X POST -H 'Content-Type: application/json' \
|
|
-d '{"messages":[{"role":"system","content":"Your prompt here"},{"role":"user","content":"Input data"}],"temperature":0,"max_tokens":300}'
|
|
```
|
|
|
|
Checklist:
|
|
- [ ] Response parses correctly (JSON keys match, LEVEL/EXPLANATION format intact)
|
|
- [ ] Error handling still works (network error, empty response)
|
|
- [ ] Existing fallbacks still trigger on failure
|
|
|
|
## Model quality expectations
|
|
|
|
1.5B models (Qwen2.5-1.5B) handle extraction and classification reasonably but:
|
|
- Priority/severity judgment is weaker than larger models
|
|
- May misclassify edge cases (2mm brake pads as "RECOMMENDED" not "CRITICAL")
|
|
- Solution: keep keyword-based fallback rules that override LLM judgments for known dangerous conditions
|
|
|
|
## Nginx prerequisite
|
|
|
|
The frontend uses a relative URL: `fetch('/llm/v1/chat/completions')`. This works because nginx serves both the static site and proxies `/llm/` to llama-server on the same port:
|
|
|
|
```nginx
|
|
location /llm/ {
|
|
proxy_pass http://127.0.0.1:8081/;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Host $host;
|
|
proxy_read_timeout 120s;
|
|
}
|
|
```
|
|
|
|
No CORS, no separate port, no API key management — the browser thinks it's calling its own server.
|