initial commit

skills/mlops/inference/llama-cpp/references/migrating-from-external-apis.md

@@ -0,0 +1,127 @@
+# Migrating from External AI APIs to llama-server
+
+Pattern: replace external LLM API calls (Gemini, OpenAI, Anthropic) with a local llama-server behind nginx.
+
+## When to use
+
+- Broken/missing API keys in a legacy codebase
+- Want to eliminate external API dependency for privacy, cost, or reliability
+- Already have llama-server running locally (see `references/deployment-patterns.md`)
+
+## Discovery: find all AI calls
+
+```bash
+# Find every fetch() or HTTP call in the project
+grep -rn 'fetch(' --include='*.js' --include='*.html' .
+# Focus on external AI APIs
+grep -rn 'googleapis\|openai.com\|api.anthropic' --include='*.js' .
+```
+
+Real-world example from an auto-repair shop web app — 3 dead Gemini calls and 1 working local LLM call already in the codebase.
+
+## Conversion recipe
+
+### Gemini → llama.cpp
+
+**Before (Gemini):**
+```javascript
+const apiKey = "AIzaSy...nZqc";
+const apiUrl = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`;
+
+const payload = {
+    contents: [{
+        role: "user",
+        parts: [{ text: prompt }]
+    }]
+};
+
+const response = await fetch(apiUrl, {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify(payload)
+});
+
+const result = await response.json();
+const text = result.candidates[0].content.parts[0].text;
+```
+
+**After (llama-server):**
+```javascript
+const response = await fetch('/llm/v1/chat/completions', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({
+        messages: [
+            {role:'system', content: prompt},       // instructions
+            {role:'user', content: userData}        // actual data
+        ],
+        temperature: 0,
+        max_tokens: 500
+    })
+});
+
+const result = await response.json();
+const text = result.choices[0].message.content;
+```
+
+### Response path mapping
+
+| Provider | Response text path |
+|---|---|
+| Gemini | `result.candidates[0].content.parts[0].text` |
+| OpenAI | `result.choices[0].message.content` |
+| llama.cpp (`/v1/chat/completions`) | `result.choices[0].message.content` |
+
+### Payload format mapping
+
+| Provider | Format |
+|---|---|
+| Gemini | `{contents: [{role: "user", parts: [{text}]}]}` |
+| OpenAI / llama.cpp | `{messages: [{role: "user", content: text}]}` |
+
+## System prompt from user prompt
+
+Gemini calls often have the full instruction text as a single `user`-role message. For llama.cpp, split it:
+
+- **System message** = the instructions/formatting rules (the "you are an expert" part)
+- **User message** = the actual data to process (service names, OCR text, etc.)
+
+This mirrors the already-working pattern in the scan screenshot feature.
+
+## Testing after migration
+
+Always test end-to-end with the actual prompt the code uses:
+
+```bash
+# Test through nginx (what the browser sees)
+curl -sk https://localhost:3447/llm/v1/chat/completions \
+  -X POST -H 'Content-Type: application/json' \
+  -d '{"messages":[{"role":"system","content":"Your prompt here"},{"role":"user","content":"Input data"}],"temperature":0,"max_tokens":300}'
+```
+
+Checklist:
+- [ ] Response parses correctly (JSON keys match, LEVEL/EXPLANATION format intact)
+- [ ] Error handling still works (network error, empty response)
+- [ ] Existing fallbacks still trigger on failure
+
+## Model quality expectations
+
+1.5B models (Qwen2.5-1.5B) handle extraction and classification reasonably but:
+- Priority/severity judgment is weaker than larger models
+- May misclassify edge cases (2mm brake pads as "RECOMMENDED" not "CRITICAL")
+- Solution: keep keyword-based fallback rules that override LLM judgments for known dangerous conditions
+
+## Nginx prerequisite
+
+The frontend uses a relative URL: `fetch('/llm/v1/chat/completions')`. This works because nginx serves both the static site and proxies `/llm/` to llama-server on the same port:
+
+```nginx
+location /llm/ {
+    proxy_pass http://127.0.0.1:8081/;
+    proxy_http_version 1.1;
+    proxy_set_header Host $host;
+    proxy_read_timeout 120s;
+}
+```
+
+No CORS, no separate port, no API key management — the browser thinks it's calling its own server.