initial commit
This commit is contained in:
ray
@@ -0,0 +1,261 @@
|
||||
---
|
||||
name: llama-cpp
|
||||
description: llama.cpp local GGUF inference + HF Hub model discovery.
|
||||
version: 2.1.2
|
||||
author: Orchestra Research
|
||||
license: MIT
|
||||
dependencies: [llama-cpp-python>=0.2.0]
|
||||
platforms: [linux, macos, windows]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [llama.cpp, GGUF, Quantization, Hugging Face Hub, CPU Inference, Apple Silicon, Edge Deployment, AMD GPUs, Intel GPUs, NVIDIA, URL-first]
|
||||
---
|
||||
|
||||
# llama.cpp + GGUF
|
||||
|
||||
Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp.
|
||||
|
||||
## When to use
|
||||
|
||||
- Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs
|
||||
- Find the right GGUF for a specific Hugging Face repo
|
||||
- Build a `llama-server` or `llama-cli` command from the Hub
|
||||
- Search the Hub for models that already support llama.cpp
|
||||
- Enumerate available `.gguf` files and sizes for a repo
|
||||
- Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM
|
||||
- **Vulkan GPU backend**: full GPU offload on NVIDIA/AMD/Intel without CUDA toolkit — no version mismatch issues, 80-90% of CUDA speed. See `references/vulkan-gpu-backend.md`
|
||||
- **Structured extraction**: parse messy semi-structured text (OCR output, logs, forms) into JSON with a small local model — see `references/structured-extraction.md`
|
||||
- **Deploy llama-server**: systemd service + nginx reverse proxy for browser-accessible inference, socket activation for VRAM-on-demand, multi-model deployment — see `references/deployment-patterns.md`
|
||||
- **Migrate from external APIs**: replace Gemini/OpenAI/Anthropic calls with local llama-server — see `references/migrating-from-external-apis.md`
|
||||
|
||||
## Model Discovery workflow
|
||||
|
||||
Prefer URL workflows before asking for `hf`, Python, or custom scripts.
|
||||
|
||||
1. Search for candidate repos on the Hub:
|
||||
- Base: `https://huggingface.co/models?apps=llama.cpp&sort=trending`
|
||||
- Add `search=<term>` for a model family
|
||||
- Add `num_parameters=min:0,max:24B` or similar when the user has size constraints
|
||||
2. Open the repo with the llama.cpp local-app view:
|
||||
- `https://huggingface.co/<repo>?local-app=llama.cpp`
|
||||
3. Treat the local-app snippet as the source of truth when it is visible:
|
||||
- copy the exact `llama-server` or `llama-cli` command
|
||||
- report the recommended quant exactly as HF shows it
|
||||
4. Read the same `?local-app=llama.cpp` URL as page text or HTML and extract the section under `Hardware compatibility`:
|
||||
- prefer its exact quant labels and sizes over generic tables
|
||||
- keep repo-specific labels such as `UD-Q4_K_M` or `IQ4_NL_XL`
|
||||
- if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance
|
||||
5. Query the tree API to confirm what actually exists:
|
||||
- `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`
|
||||
- keep entries where `type` is `file` and `path` ends with `.gguf`
|
||||
- use `path` and `size` as the source of truth for filenames and byte sizes
|
||||
- separate quantized checkpoints from `mmproj-*.gguf` projector files and `BF16/` shard files
|
||||
- use `https://huggingface.co/<repo>/tree/main` only as a human fallback
|
||||
6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant:
|
||||
- shorthand quant selection: `llama-server -hf <repo>:<QUANT>`
|
||||
- exact-file fallback: `llama-server --hf-repo <repo> --hf-file <filename.gguf>`
|
||||
7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.
|
||||
|
||||
## Quick start
|
||||
|
||||
### Install llama.cpp
|
||||
|
||||
```bash
|
||||
# macOS / Linux (simplest)
|
||||
brew install llama.cpp
|
||||
```
|
||||
|
||||
```bash
|
||||
winget install llama.cpp
|
||||
```
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp
|
||||
cd llama.cpp
|
||||
cmake -B build
|
||||
cmake --build build --config Release
|
||||
```
|
||||
|
||||
### Run directly from the Hugging Face Hub
|
||||
|
||||
```bash
|
||||
llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
|
||||
```
|
||||
|
||||
```bash
|
||||
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
|
||||
```
|
||||
|
||||
### Run an exact GGUF file from the Hub
|
||||
|
||||
Use this when the tree API shows custom file naming or the exact HF snippet is missing.
|
||||
|
||||
```bash
|
||||
llama-server \
|
||||
--hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
|
||||
--hf-file Phi-3-mini-4k-instruct-q4.gguf \
|
||||
-c 4096
|
||||
```
|
||||
|
||||
### OpenAI-compatible server check
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [
|
||||
{"role": "user", "content": "Write a limerick about Python exceptions"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Python bindings (llama-cpp-python)
|
||||
|
||||
`pip install llama-cpp-python` (CUDA: `CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir`; Metal: `CMAKE_ARGS="-DGGML_METAL=on" ...`).
|
||||
|
||||
### Basic generation
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="./model-q4_k_m.gguf",
|
||||
n_ctx=4096,
|
||||
n_gpu_layers=35, # 0 for CPU, 99 to offload everything
|
||||
n_threads=8,
|
||||
)
|
||||
|
||||
out = llm("What is machine learning?", max_tokens=256, temperature=0.7)
|
||||
print(out["choices"][0]["text"])
|
||||
```
|
||||
|
||||
### Chat + streaming
|
||||
|
||||
```python
|
||||
llm = Llama(
|
||||
model_path="./model-q4_k_m.gguf",
|
||||
n_ctx=4096,
|
||||
n_gpu_layers=35,
|
||||
chat_format="llama-3", # or "chatml", "mistral", etc.
|
||||
)
|
||||
|
||||
resp = llm.create_chat_completion(
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant."},
|
||||
{"role": "user", "content": "What is Python?"},
|
||||
],
|
||||
max_tokens=256,
|
||||
)
|
||||
print(resp["choices"][0]["message"]["content"])
|
||||
|
||||
# Streaming
|
||||
for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True):
|
||||
print(chunk["choices"][0]["text"], end="", flush=True)
|
||||
```
|
||||
|
||||
### Embeddings
|
||||
|
||||
```python
|
||||
llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35)
|
||||
vec = llm.embed("This is a test sentence.")
|
||||
print(f"Embedding dimension: {len(vec)}")
|
||||
```
|
||||
|
||||
You can also load a GGUF straight from the Hub:
|
||||
|
||||
```python
|
||||
llm = Llama.from_pretrained(
|
||||
repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",
|
||||
filename="*Q4_K_M.gguf",
|
||||
n_gpu_layers=35,
|
||||
)
|
||||
```
|
||||
|
||||
## Choosing a quant
|
||||
|
||||
Use the Hub page first, generic heuristics second.
|
||||
|
||||
- Prefer the exact quant that HF marks as compatible for the user's hardware profile.
|
||||
- For general chat, start with `Q4_K_M`.
|
||||
- For code or technical work, prefer `Q5_K_M` or `Q6_K` if memory allows.
|
||||
- For very tight RAM budgets, consider `Q3_K_M`, `IQ` variants, or `Q2` variants only if the user explicitly prioritizes fit over quality.
|
||||
- For multimodal repos, mention `mmproj-*.gguf` separately. The projector is not the main model file.
|
||||
- Do not normalize repo-native labels. If the page says `UD-Q4_K_M`, report `UD-Q4_K_M`.
|
||||
|
||||
## Extracting available GGUFs from a repo
|
||||
|
||||
When the user asks what GGUFs exist, return:
|
||||
|
||||
- filename
|
||||
- file size
|
||||
- quant label
|
||||
- whether it is a main model or an auxiliary projector
|
||||
|
||||
Ignore unless requested:
|
||||
|
||||
- README
|
||||
- BF16 shard files
|
||||
- imatrix blobs or calibration artifacts
|
||||
|
||||
Use the tree API for this step:
|
||||
|
||||
- `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`
|
||||
|
||||
For a repo like `unsloth/Qwen3.6-35B-A3B-GGUF`, the local-app page can show quant chips such as `UD-Q4_K_M`, `UD-Q5_K_M`, `UD-Q6_K`, and `Q8_0`, while the tree API exposes exact file paths such as `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` and `Qwen3.6-35B-A3B-Q8_0.gguf` with byte sizes. Use the tree API to turn a quant label into an exact filename.
|
||||
|
||||
## Search patterns
|
||||
|
||||
Use these URL shapes directly:
|
||||
|
||||
```text
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
https://huggingface.co/<repo>/tree/main
|
||||
```
|
||||
|
||||
## Output format
|
||||
|
||||
When answering discovery requests, prefer a compact structured result like:
|
||||
|
||||
```text
|
||||
Repo: <repo>
|
||||
Recommended quant from HF: <label> (<size>)
|
||||
llama-server: <command>
|
||||
Other GGUFs:
|
||||
- <filename> - <size>
|
||||
- <filename> - <size>
|
||||
Source URLs:
|
||||
- <local-app URL>
|
||||
- <tree API URL>
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
> This skill absorbs the former `local-model-management` and `hermes-local-inference` skills. Their unique content — Ollama operations, Hermes config wiring for local models, auxiliary section mapping, and GPU investigation — is now consolidated here and in the new references below.
|
||||
|
||||
- **[vulkan-gpu-backend.md](references/vulkan-gpu-backend.md)** — build llama.cpp with Vulkan for GPU offload on NVIDIA/AMD/Intel without CUDA toolkit; VRAM sizing, systemd service, detection
|
||||
- **[gpu-hardware-recommendations.md](references/gpu-hardware-recommendations.md)** — consumer GPU comparison for llama.cpp inference; NVIDIA vs AMD, VRAM/bandwidth tradeoffs, CUDA vs Vulkan, power requirements, system profiling commands
|
||||
- **[hub-discovery.md](references/hub-discovery.md)** — URL-only Hugging Face workflows, search patterns, GGUF extraction, and command reconstruction
|
||||
- **[deployment-patterns.md](references/deployment-patterns.md)** — systemd service template, nginx reverse proxy, browser integration, OCR-to-LLM pipeline, prompt engineering for structured extraction
|
||||
- **[migrating-from-external-apis.md](references/migrating-from-external-apis.md)** — replacing Gemini/OpenAI API calls with local llama-server; payload conversion, response path mapping, testing checklist
|
||||
- **[structured-extraction.md](references/structured-extraction.md)** — prompt template for JSON extraction from messy OCR text with small models, field normalization, fallback patterns, Tesseract preprocessing
|
||||
- **[advanced-usage.md](references/advanced-usage.md)** — speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU, custom builds, benchmark scripts
|
||||
- **[quantization.md](references/quantization.md)** — quant quality tradeoffs, when to use Q4/Q5/Q6/IQ, model size scaling, imatrix
|
||||
- **[server.md](references/server.md)** — direct-from-Hub server launch, OpenAI API endpoints, Docker deployment, NGINX load balancing, monitoring
|
||||
- **[optimization.md](references/optimization.md)** — CPU threading, BLAS, GPU offload heuristics, batch tuning, benchmarks
|
||||
- **[ollama-deployment.md](references/ollama-deployment.md)** — Ollama as an alternative to standalone llama-server; nginx Host header fix, on-demand VRAM unloading, vision model API, migrating from llama-server, replacing Tesseract OCR with a VLM
|
||||
- **[vram-sizing.md](references/vram-sizing.md)** — model size reference table, quantization multipliers, known-good GPU+model combinations, GPU investigation commands, removing stale servers (absorbed from `hermes-local-inference` and `local-model-management`)
|
||||
- **[hermes-ollama-config.md](references/hermes-ollama-config.md)** — Hermes Agent configuration for Ollama provider: main model, all auxiliary sections, gateway restart, and switching back to cloud (absorbed from `hermes-local-inference` and `local-model-management`)
|
||||
## Resources
|
||||
|
||||
- **GitHub**: https://github.com/ggml-org/llama.cpp
|
||||
- **Hugging Face GGUF + llama.cpp docs**: https://huggingface.co/docs/hub/gguf-llamacpp
|
||||
- **Hugging Face Local Apps docs**: https://huggingface.co/docs/hub/main/local-apps
|
||||
- **Hugging Face Local Agents docs**: https://huggingface.co/docs/hub/agents-local
|
||||
- **Example local-app page**: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
|
||||
- **Example tree API**: https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
|
||||
- **Example llama.cpp search**: https://huggingface.co/models?num_parameters=min:0,max:24B&apps=llama.cpp&sort=trending
|
||||
- **License**: MIT
|
||||
@@ -0,0 +1,504 @@
|
||||
# GGUF Advanced Usage Guide
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
### Draft Model Approach
|
||||
|
||||
```bash
|
||||
# Use smaller model as draft for faster generation
|
||||
./llama-speculative \
|
||||
-m large-model-q4_k_m.gguf \
|
||||
-md draft-model-q4_k_m.gguf \
|
||||
-p "Write a story about AI" \
|
||||
-n 500 \
|
||||
--draft 8 # Draft tokens before verification
|
||||
```
|
||||
|
||||
### Self-Speculative Decoding
|
||||
|
||||
```bash
|
||||
# Use same model with different context for speculation
|
||||
./llama-cli -m model-q4_k_m.gguf \
|
||||
--lookup-cache-static lookup.bin \
|
||||
--lookup-cache-dynamic lookup-dynamic.bin \
|
||||
-p "Hello world"
|
||||
```
|
||||
|
||||
## Batched Inference
|
||||
|
||||
### Process Multiple Prompts
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
n_ctx=4096,
|
||||
n_gpu_layers=35,
|
||||
n_batch=512 # Larger batch for parallel processing
|
||||
)
|
||||
|
||||
prompts = [
|
||||
"What is Python?",
|
||||
"Explain machine learning.",
|
||||
"Describe neural networks."
|
||||
]
|
||||
|
||||
# Process in batch (each prompt gets separate context)
|
||||
for prompt in prompts:
|
||||
output = llm(prompt, max_tokens=100)
|
||||
print(f"Q: {prompt}")
|
||||
print(f"A: {output['choices'][0]['text']}\n")
|
||||
```
|
||||
|
||||
### Server Batching
|
||||
|
||||
```bash
|
||||
# Start server with batching
|
||||
./llama-server -m model-q4_k_m.gguf \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080 \
|
||||
-ngl 35 \
|
||||
-c 4096 \
|
||||
--parallel 4 # Concurrent requests
|
||||
--cont-batching # Continuous batching
|
||||
```
|
||||
|
||||
## Custom Model Conversion
|
||||
|
||||
### Convert with Vocabulary Modifications
|
||||
|
||||
```python
|
||||
# custom_convert.py
|
||||
import sys
|
||||
sys.path.insert(0, './llama.cpp')
|
||||
|
||||
from convert_hf_to_gguf import main
|
||||
from gguf import GGUFWriter
|
||||
|
||||
# Custom conversion with modified vocab
|
||||
def convert_with_custom_vocab(model_path, output_path):
|
||||
# Load and modify tokenizer
|
||||
from transformers import AutoTokenizer
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
||||
|
||||
# Add special tokens if needed
|
||||
special_tokens = {"additional_special_tokens": ["<|custom|>"]}
|
||||
tokenizer.add_special_tokens(special_tokens)
|
||||
tokenizer.save_pretrained(model_path)
|
||||
|
||||
# Then run standard conversion
|
||||
main([model_path, "--outfile", output_path])
|
||||
```
|
||||
|
||||
### Convert Specific Architecture
|
||||
|
||||
```bash
|
||||
# For Mistral-style models
|
||||
python convert_hf_to_gguf.py ./mistral-model \
|
||||
--outfile mistral-f16.gguf \
|
||||
--outtype f16
|
||||
|
||||
# For Qwen models
|
||||
python convert_hf_to_gguf.py ./qwen-model \
|
||||
--outfile qwen-f16.gguf \
|
||||
--outtype f16
|
||||
|
||||
# For Phi models
|
||||
python convert_hf_to_gguf.py ./phi-model \
|
||||
--outfile phi-f16.gguf \
|
||||
--outtype f16
|
||||
```
|
||||
|
||||
## Advanced Quantization
|
||||
|
||||
### Mixed Quantization
|
||||
|
||||
```bash
|
||||
# Quantize different layer types differently
|
||||
./llama-quantize model-f16.gguf model-mixed.gguf Q4_K_M \
|
||||
--allow-requantize \
|
||||
--leave-output-tensor
|
||||
```
|
||||
|
||||
### Quantization with Token Embeddings
|
||||
|
||||
```bash
|
||||
# Keep embeddings at higher precision
|
||||
./llama-quantize model-f16.gguf model-q4.gguf Q4_K_M \
|
||||
--token-embedding-type f16
|
||||
```
|
||||
|
||||
### IQ Quantization (Importance-aware)
|
||||
|
||||
```bash
|
||||
# Ultra-low bit quantization with importance
|
||||
./llama-quantize --imatrix model.imatrix \
|
||||
model-f16.gguf model-iq2_xxs.gguf IQ2_XXS
|
||||
|
||||
# Available IQ types: IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_XS, IQ3_S, IQ4_XS
|
||||
```
|
||||
|
||||
## Memory Optimization
|
||||
|
||||
### Memory Mapping
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
# Use memory mapping for large models
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
use_mmap=True, # Memory map the model
|
||||
use_mlock=False, # Don't lock in RAM
|
||||
n_gpu_layers=35
|
||||
)
|
||||
```
|
||||
|
||||
### Partial GPU Offload
|
||||
|
||||
```python
|
||||
# Calculate layers to offload based on VRAM
|
||||
import subprocess
|
||||
|
||||
def get_free_vram_gb():
|
||||
result = subprocess.run(
|
||||
['nvidia-smi', '--query-gpu=memory.free', '--format=csv,nounits,noheader'],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
return int(result.stdout.strip()) / 1024
|
||||
|
||||
# Estimate layers based on VRAM (rough: 0.5GB per layer for 7B Q4)
|
||||
free_vram = get_free_vram_gb()
|
||||
layers_to_offload = int(free_vram / 0.5)
|
||||
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
n_gpu_layers=min(layers_to_offload, 35) # Cap at total layers
|
||||
)
|
||||
```
|
||||
|
||||
### KV Cache Optimization
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
# Optimize KV cache for long contexts
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
n_ctx=8192, # Large context
|
||||
n_gpu_layers=35,
|
||||
type_k=1, # Q8_0 for K cache (1)
|
||||
type_v=1, # Q8_0 for V cache (1)
|
||||
# Or use Q4_0 (2) for more compression
|
||||
)
|
||||
```
|
||||
|
||||
## Context Management
|
||||
|
||||
### Context Shifting
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
n_ctx=4096,
|
||||
n_gpu_layers=35
|
||||
)
|
||||
|
||||
# Handle long conversations with context shifting
|
||||
conversation = []
|
||||
max_history = 10
|
||||
|
||||
def chat(user_message):
|
||||
conversation.append({"role": "user", "content": user_message})
|
||||
|
||||
# Keep only recent history
|
||||
if len(conversation) > max_history * 2:
|
||||
conversation = conversation[-max_history * 2:]
|
||||
|
||||
response = llm.create_chat_completion(
|
||||
messages=conversation,
|
||||
max_tokens=256
|
||||
)
|
||||
|
||||
assistant_message = response["choices"][0]["message"]["content"]
|
||||
conversation.append({"role": "assistant", "content": assistant_message})
|
||||
return assistant_message
|
||||
```
|
||||
|
||||
### Save and Load State
|
||||
|
||||
```bash
|
||||
# Save state to file
|
||||
./llama-cli -m model.gguf \
|
||||
-p "Once upon a time" \
|
||||
--save-session session.bin \
|
||||
-n 100
|
||||
|
||||
# Load and continue
|
||||
./llama-cli -m model.gguf \
|
||||
--load-session session.bin \
|
||||
-p " and they lived" \
|
||||
-n 100
|
||||
```
|
||||
|
||||
## Grammar Constrained Generation
|
||||
|
||||
### JSON Output
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama, LlamaGrammar
|
||||
|
||||
# Define JSON grammar
|
||||
json_grammar = LlamaGrammar.from_string('''
|
||||
root ::= object
|
||||
object ::= "{" ws pair ("," ws pair)* "}" ws
|
||||
pair ::= string ":" ws value
|
||||
value ::= string | number | object | array | "true" | "false" | "null"
|
||||
array ::= "[" ws value ("," ws value)* "]" ws
|
||||
string ::= "\\"" [^"\\\\]* "\\""
|
||||
number ::= [0-9]+
|
||||
ws ::= [ \\t\\n]*
|
||||
''')
|
||||
|
||||
llm = Llama(model_path="model-q4_k_m.gguf", n_gpu_layers=35)
|
||||
|
||||
output = llm(
|
||||
"Output a JSON object with name and age:",
|
||||
grammar=json_grammar,
|
||||
max_tokens=100
|
||||
)
|
||||
print(output["choices"][0]["text"])
|
||||
```
|
||||
|
||||
### Custom Grammar
|
||||
|
||||
```python
|
||||
# Grammar for specific format
|
||||
answer_grammar = LlamaGrammar.from_string('''
|
||||
root ::= "Answer: " letter "\\n" "Explanation: " explanation
|
||||
letter ::= [A-D]
|
||||
explanation ::= [a-zA-Z0-9 .,!?]+
|
||||
''')
|
||||
|
||||
output = llm(
|
||||
"Q: What is 2+2? A) 3 B) 4 C) 5 D) 6",
|
||||
grammar=answer_grammar,
|
||||
max_tokens=100
|
||||
)
|
||||
```
|
||||
|
||||
## LoRA Integration
|
||||
|
||||
### Load LoRA Adapter
|
||||
|
||||
```bash
|
||||
# Apply LoRA at runtime
|
||||
./llama-cli -m base-model-q4_k_m.gguf \
|
||||
--lora lora-adapter.gguf \
|
||||
--lora-scale 1.0 \
|
||||
-p "Hello!"
|
||||
```
|
||||
|
||||
### Multiple LoRA Adapters
|
||||
|
||||
```bash
|
||||
# Stack multiple adapters
|
||||
./llama-cli -m base-model.gguf \
|
||||
--lora adapter1.gguf --lora-scale 0.5 \
|
||||
--lora adapter2.gguf --lora-scale 0.5 \
|
||||
-p "Hello!"
|
||||
```
|
||||
|
||||
### Python LoRA Usage
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="base-model-q4_k_m.gguf",
|
||||
lora_path="lora-adapter.gguf",
|
||||
lora_scale=1.0,
|
||||
n_gpu_layers=35
|
||||
)
|
||||
```
|
||||
|
||||
## Embedding Generation
|
||||
|
||||
### Extract Embeddings
|
||||
|
||||
```python
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="model-q4_k_m.gguf",
|
||||
embedding=True, # Enable embedding mode
|
||||
n_gpu_layers=35
|
||||
)
|
||||
|
||||
# Get embeddings
|
||||
embeddings = llm.embed("This is a test sentence.")
|
||||
print(f"Embedding dimension: {len(embeddings)}")
|
||||
```
|
||||
|
||||
### Batch Embeddings
|
||||
|
||||
```python
|
||||
texts = [
|
||||
"Machine learning is fascinating.",
|
||||
"Deep learning uses neural networks.",
|
||||
"Python is a programming language."
|
||||
]
|
||||
|
||||
embeddings = [llm.embed(text) for text in texts]
|
||||
|
||||
# Calculate similarity
|
||||
import numpy as np
|
||||
|
||||
def cosine_similarity(a, b):
|
||||
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
|
||||
|
||||
sim = cosine_similarity(embeddings[0], embeddings[1])
|
||||
print(f"Similarity: {sim:.4f}")
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Benchmark Script
|
||||
|
||||
```python
|
||||
import time
|
||||
from llama_cpp import Llama
|
||||
|
||||
def benchmark(model_path, prompt, n_tokens=100, n_runs=5):
|
||||
llm = Llama(
|
||||
model_path=model_path,
|
||||
n_gpu_layers=35,
|
||||
n_ctx=2048,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Warmup
|
||||
llm(prompt, max_tokens=10)
|
||||
|
||||
# Benchmark
|
||||
times = []
|
||||
for _ in range(n_runs):
|
||||
start = time.time()
|
||||
output = llm(prompt, max_tokens=n_tokens)
|
||||
elapsed = time.time() - start
|
||||
times.append(elapsed)
|
||||
|
||||
avg_time = sum(times) / len(times)
|
||||
tokens_per_sec = n_tokens / avg_time
|
||||
|
||||
print(f"Model: {model_path}")
|
||||
print(f"Avg time: {avg_time:.2f}s")
|
||||
print(f"Tokens/sec: {tokens_per_sec:.1f}")
|
||||
|
||||
return tokens_per_sec
|
||||
|
||||
# Compare quantizations
|
||||
for quant in ["q4_k_m", "q5_k_m", "q8_0"]:
|
||||
benchmark(f"model-{quant}.gguf", "Explain quantum computing:", 100)
|
||||
```
|
||||
|
||||
### Optimal Configuration Finder
|
||||
|
||||
```python
|
||||
def find_optimal_config(model_path, target_vram_gb=8):
|
||||
"""Find optimal n_gpu_layers and n_batch for target VRAM."""
|
||||
from llama_cpp import Llama
|
||||
import gc
|
||||
|
||||
best_config = None
|
||||
best_speed = 0
|
||||
|
||||
for n_gpu_layers in range(0, 50, 5):
|
||||
for n_batch in [128, 256, 512, 1024]:
|
||||
try:
|
||||
gc.collect()
|
||||
llm = Llama(
|
||||
model_path=model_path,
|
||||
n_gpu_layers=n_gpu_layers,
|
||||
n_batch=n_batch,
|
||||
n_ctx=2048,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Quick benchmark
|
||||
start = time.time()
|
||||
llm("Hello", max_tokens=50)
|
||||
speed = 50 / (time.time() - start)
|
||||
|
||||
if speed > best_speed:
|
||||
best_speed = speed
|
||||
best_config = {
|
||||
"n_gpu_layers": n_gpu_layers,
|
||||
"n_batch": n_batch,
|
||||
"speed": speed
|
||||
}
|
||||
|
||||
del llm
|
||||
gc.collect()
|
||||
|
||||
except Exception as e:
|
||||
print(f"OOM at layers={n_gpu_layers}, batch={n_batch}")
|
||||
break
|
||||
|
||||
return best_config
|
||||
```
|
||||
|
||||
## Multi-GPU Setup
|
||||
|
||||
### Distribute Across GPUs
|
||||
|
||||
```bash
|
||||
# Split model across multiple GPUs
|
||||
./llama-cli -m large-model.gguf \
|
||||
--tensor-split 0.5,0.5 \
|
||||
-ngl 60 \
|
||||
-p "Hello!"
|
||||
```
|
||||
|
||||
### Python Multi-GPU
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
|
||||
|
||||
from llama_cpp import Llama
|
||||
|
||||
llm = Llama(
|
||||
model_path="large-model-q4_k_m.gguf",
|
||||
n_gpu_layers=60,
|
||||
tensor_split=[0.5, 0.5] # Split evenly across 2 GPUs
|
||||
)
|
||||
```
|
||||
|
||||
## Custom Builds
|
||||
|
||||
### Build with All Optimizations
|
||||
|
||||
```bash
|
||||
# Clean build with all CPU optimizations
|
||||
make clean
|
||||
LLAMA_OPENBLAS=1 LLAMA_BLAS_VENDOR=OpenBLAS make -j
|
||||
|
||||
# With CUDA and cuBLAS
|
||||
make clean
|
||||
GGML_CUDA=1 LLAMA_CUBLAS=1 make -j
|
||||
|
||||
# With specific CUDA architecture
|
||||
GGML_CUDA=1 CUDA_DOCKER_ARCH=sm_86 make -j
|
||||
```
|
||||
|
||||
### CMake Build
|
||||
|
||||
```bash
|
||||
mkdir build && cd build
|
||||
cmake .. -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build . --config Release -j
|
||||
```
|
||||
@@ -0,0 +1,247 @@
|
||||
# llama.cpp Deployment Patterns
|
||||
|
||||
## CPU-only build (CUDA toolkit mismatch workaround)
|
||||
|
||||
When nvcc and CUDA libraries are from different versions (e.g., nvcc 12.4, libcublas.so from CUDA 13.1), the CUDA build will fail with linker errors like `undefined reference to cublasGemmEx@libcublas.so.12`.
|
||||
|
||||
**Fix:** Build CPU-only. For models ≤3B parameters, CPU inference is fast enough (20-40 tok/s).
|
||||
|
||||
```bash
|
||||
git clone --depth 1 --branch <tag> https://github.com/ggml-org/llama.cpp.git
|
||||
cd llama.cpp
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --config Release -j4
|
||||
```
|
||||
|
||||
## Systemd service
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=llama.cpp server for <model>
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=<user>
|
||||
ExecStart=<path>/llama-server \
|
||||
-m <path>/model.gguf \
|
||||
-c 4096 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8081 \
|
||||
-t 4
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Install:
|
||||
```bash
|
||||
sudo cp llama-server.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now llama-server
|
||||
```
|
||||
|
||||
## Nginx reverse proxy
|
||||
|
||||
For browser access to llama-server (avoids CORS, works across devices):
|
||||
|
||||
```nginx
|
||||
location /llm/ {
|
||||
proxy_pass http://127.0.0.1:8081/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_read_timeout 120s;
|
||||
}
|
||||
```
|
||||
|
||||
Test: `curl -sk https://your-domain:port/llm/health`
|
||||
|
||||
## Browser integration
|
||||
|
||||
```javascript
|
||||
const resp = await fetch('/llm/v1/chat/completions', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
messages: [
|
||||
{role:'system', content:'Your prompt'},
|
||||
{role:'user', content: userText}
|
||||
],
|
||||
temperature: 0,
|
||||
max_tokens: 500
|
||||
})
|
||||
});
|
||||
const data = await resp.json();
|
||||
let raw = data.choices[0].message.content;
|
||||
// Strip markdown fences (small models often wrap JSON)
|
||||
raw = raw.replace(/^```(?:json)?\s*\n?/i, '').replace(/\n?```\s*$/i, '');
|
||||
const parsed = JSON.parse(raw);
|
||||
```
|
||||
|
||||
## OCR → LLM pipeline pattern
|
||||
|
||||
1. Tesseract.js runs OCR in browser → raw text
|
||||
2. Browser POSTs text to `/llm/v1/chat/completions`
|
||||
3. llama-server processes → returns structured JSON
|
||||
4. Client normalizes fields (strip phone dashes, uppercase VIN)
|
||||
5. Fallback to rule-based parser if LLM unavailable
|
||||
|
||||
## Prompt engineering for structured extraction
|
||||
|
||||
Key rules for reliable JSON extraction from small models (1.5B-3B):
|
||||
- Return ONLY JSON, no markdown — but strip fences anyway
|
||||
- Explicit field rules in system prompt (not just schema)
|
||||
- Include example values in field descriptions
|
||||
- Use temperature=0 for deterministic output
|
||||
- `max_tokens=500` sufficient for 3-5 appointment records
|
||||
|
||||
Example system prompt for OCR appointment parsing:
|
||||
```
|
||||
Extract appointment details from OCR text. Return ONLY valid JSON, no markdown.
|
||||
|
||||
RULES:
|
||||
- customerName: person name before phone number. Strip "RO" prefix.
|
||||
- customerPhone: 10 digits only, no dashes
|
||||
- customerEmail: actual email with @
|
||||
- vin: 17-char uppercase VIN. OCR may misread 0 as O, 1 as I.
|
||||
- vehicleInfo: year + make + model
|
||||
- serviceType: work description after opcode
|
||||
- opCode: only bracketed code like [REP] — no advisor names
|
||||
- appointmentTime: 24h format (e.g. "09:00")
|
||||
- duration: integer minutes
|
||||
|
||||
JSON: {"appointments":[{"customerName":"","customerPhone":"","customerEmail":"","vin":"","vehicleInfo":"","serviceType":"","opCode":"","appointmentTime":"","duration":0}]}
|
||||
```
|
||||
|
||||
## Socket-activated llama-server (VRAM on-demand)
|
||||
|
||||
When the GPU is shared with other workloads (gaming, other models), keeping the model in VRAM 24/7 wastes resources. systemd socket activation starts llama-server only when a request hits the port, and auto-stops it after idle timeout.
|
||||
|
||||
**Cold-start latency:** 7-8s for a ~9 GB GGUF on PCIe 3.0 + NVMe (NVMe read ~3.5s, PCIe transfer ~0.7s, CUDA init ~3s). Warm restart (file cached in RAM): 3-4s. Acceptable for batch OCR and non-interactive use.
|
||||
|
||||
### Socket file
|
||||
|
||||
```ini
|
||||
# /etc/systemd/system/llama-server.socket
|
||||
[Socket]
|
||||
ListenStream=127.0.0.1:8081
|
||||
|
||||
[Install]
|
||||
WantedBy=sockets.target
|
||||
```
|
||||
|
||||
### Service file (modified)
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=llama.cpp server (socket-activated)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ray
|
||||
WorkingDirectory=/home/ray
|
||||
ExecStart=/path/to/llama-server \
|
||||
-m /path/to/model.gguf \
|
||||
-c 4096 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8081 \
|
||||
-ngl 99 \
|
||||
-t 8
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
StopIdleSec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
**`StopIdleSec=300`** — systemd kills the service after 5 minutes with no active connections. Adjust up/down as needed.
|
||||
|
||||
### Activation
|
||||
|
||||
```bash
|
||||
sudo systemctl stop llama-server
|
||||
sudo systemctl disable llama-server
|
||||
sudo systemctl enable --now llama-server.socket
|
||||
# Now llama-server only launches when :8081 receives a connection
|
||||
```
|
||||
|
||||
**Verification:** `ss -tlnp | grep 8081` shows the socket in LISTEN state with systemd as the listener. First request triggers service start.
|
||||
|
||||
### VRAM lifecycle
|
||||
|
||||
| State | VRAM |
|
||||
|---|---|
|
||||
| Idle (no connections) | 0 MB (service not running) |
|
||||
| Active (handling requests) | Model size (~3-9 GB depending on quant) |
|
||||
| Post-idle (StopIdleSec elapsed) | 0 MB (service killed, VRAM freed) |
|
||||
|
||||
### When NOT to socket-activate
|
||||
|
||||
- Interactive chat where 7-8s cold start is annoying
|
||||
- Frequent bursts of requests (model reloads repeatedly)
|
||||
- When the GPU has enough VRAM to leave the model resident permanently (e.g., 24 GB card with a 7B model)
|
||||
|
||||
## Multi-model deployment
|
||||
|
||||
Running two llama-server instances on different ports for different purposes (e.g., small model for chat + larger model for delegation/coding).
|
||||
|
||||
```bash
|
||||
# Instance 1: main model (port 8081)
|
||||
llama-server -m Qwen2.5-7B-Instruct-Q4_K_M.gguf -c 4096 --port 8081 -ngl 99 -t 8 &
|
||||
|
||||
# Instance 2: delegation/coding model (port 8082)
|
||||
llama-server -m Qwen2.5-Coder-14B-Instruct-Q4_K_M.gguf -c 16384 --port 8082 -ngl 99 -t 8 &
|
||||
```
|
||||
|
||||
### VRAM budgeting for dual models
|
||||
|
||||
Must fit within total GPU VRAM. Example on RTX 2080 Ti (11 GB):
|
||||
|
||||
| Combination | VRAM used | Fits? |
|
||||
|---|---|---|
|
||||
| 7B Q2_K (3.2 GB) + 7B Q4_K_M (4.7 GB) | ~7.9 GB | ✓ |
|
||||
| 7B Q2_K (3.2 GB) + 14B Q4_K_M (9 GB) | ~12.2 GB | ✗ |
|
||||
| 3B Q4_K_M (2 GB) + 14B Q4_K_M (9 GB) | ~11 GB | Tight |
|
||||
|
||||
For dual-model on a single GPU, prefer socket activation on one instance so they don't both stay loaded. Or accept that only one runs at a time.
|
||||
|
||||
### Hermes delegation config
|
||||
|
||||
Point Hermes subagents at a local delegation model:
|
||||
|
||||
```yaml
|
||||
# ~/.hermes/config.yaml
|
||||
delegation:
|
||||
provider: custom:local-delegation
|
||||
model: qwen-coder-14b
|
||||
base_url: http://127.0.0.1:8082/v1
|
||||
api_key: not-needed
|
||||
max_concurrent_children: 1 # local model can't parallelize well
|
||||
```
|
||||
|
||||
Any OpenAI-compatible client can be pointed at llama-server the same way — set `base_url` to `http://127.0.0.1:<port>/v1` and use a placeholder API key.
|
||||
|
||||
### Quality expectations for delegation
|
||||
|
||||
A local 14B Q4 model vs a cloud API (e.g., DeepSeek V4 Flash):
|
||||
|
||||
| Dimension | Local 14B Q4 | Cloud API |
|
||||
|---|---|---|
|
||||
| Structured tasks (file ops, patches, grep) | Good | Excellent |
|
||||
| Simple debugging | Good | Excellent |
|
||||
| Complex multi-step reasoning | Fair | Excellent |
|
||||
| Code generation (new features) | Good | Very good |
|
||||
| Cost per delegation | $0 | API tokens |
|
||||
| Latency | 45-55 tok/s local | API round-trip |
|
||||
|
||||
For delegation workloads (which are mostly structured), a 14B at Q4 is competent. The quality gap is real but often acceptable for the cost savings.
|
||||
|
||||
- GPU: GTX 1050 Ti 4GB VRAM
|
||||
- RAM: 14GB
|
||||
- Qwen2.5-1.5B-Instruct Q4_K_M: 941 MB, ~30 tok/s CPU, loads in ~500ms
|
||||
- Llama-3.2-3B-Instruct Q4_K_M: ~2 GB, would fit in VRAM if CUDA worked
|
||||
@@ -0,0 +1,125 @@
|
||||
# GPU Hardware Recommendations for llama.cpp
|
||||
|
||||
Hardware analysis for selecting a GPU for local llama.cpp inference. All analysis assumes llama.cpp as the inference engine.
|
||||
|
||||
## The Golden Rule: CUDA is Everything
|
||||
|
||||
For llama.cpp inference, **NVIDIA CUDA is the only sane choice for consumer GPUs**. AMD consumer cards (RDNA2/RDNA3) are locked to Vulkan backend — no ROCm support on consumer SKUs. The Vulkan backend works but has real limitations:
|
||||
|
||||
- No flash attention
|
||||
- No MMQ kernels (slower quants)
|
||||
- More bugs, less optimization priority from the llama.cpp team
|
||||
- Real-world token generation is often on par with or slower than a weaker NVIDIA card on CUDA, despite higher raw bandwidth
|
||||
|
||||
**AMD integrated GPUs and iGPUs**: Vulkan works fine for these (they're small anyway). The problem is discrete AMD consumer cards where you're paying for bandwidth you can't fully use.
|
||||
|
||||
## The 128-Bit Bus Trap
|
||||
|
||||
NVIDIA's xx60 series (4060, 4060 Ti, 5060, 5060 Ti) uses a crippled **128-bit memory bus** — half the width of older cards like the GTX 1080 (256-bit) or RTX 2080 Ti (352-bit). Even with GDDR6X/GDDR7 speeds, the narrow bus caps effective bandwidth so severely that a $400 RTX 4060 Ti 16GB (288 GB/s) is **slower than an $80 GTX 1080 (320 GB/s)** for LLM token generation.
|
||||
|
||||
**The 16GB dead zone**: the first NVIDIA 16GB card with a bus wide enough for LLMs is the RTX 4070 Ti Super (256-bit, 672 GB/s) at ~$600. Every 16GB card below it is a 128-bit gaming card masquerading as an AI card — attractive VRAM number, useless bandwidth.
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| GPU | VRAM | Bandwidth | Bus | CUDA | Price (used) | Best model fits | Notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| GTX 1050 Ti | 4 GB | 112 GB/s | 128-bit | ❌ Vulkan only | — | Q2_K 7B | Slot-powered. Baseline. |
|
||||
| GTX 1070 | 8 GB | 256 GB/s | 256-bit | ✅ | ~$80-100 | Q4_K_M 7B, Q2 14B | 1× 8-pin, 150W. |
|
||||
| **GTX 1080** | 8 GB | 320 GB/s | 256-bit | ✅ | **~$80** | Q5_K_M 7B | Best budget LLM card. Faster than 4060 Ti 16GB. |
|
||||
| GTX 1080 Ti | 11 GB | 484 GB/s | 352-bit | ✅ | ~$140-170 | Q4_K_M 14B | Sweet spot. 11GB + 484 GB/s. |
|
||||
| **RTX 2080 Ti** | 11 GB | 616 GB/s | 352-bit | ✅ + Tensor | ~$250-300 | Q4_K_M 14B, IQ3 20B | Best sub-$300 card. 2× RTX 2060 12GB speed. |
|
||||
| RTX 2060 12GB | 12 GB | 336 GB/s | 192-bit | ✅ + Tensor | ~$200 | Q4_K_M 14B | 1GB more than 1080 Ti but 44% slower. VLMs need 12GB+. |
|
||||
| RTX 2060 SUPER | 8 GB | 448 GB/s | 256-bit | ✅ + Tensor | ~$180-220 | Q5_K_M 7B | Fast 7B card, hard-capped at 8GB. |
|
||||
| RTX 3070 Ti | 8 GB | 608 GB/s | 256-bit | ✅ + Tensor | ~$250-300 | Q5_K_M 7B | Very fast, but 8GB only — same ceiling as $80 GTX 1080. |
|
||||
| RTX 3080 10GB | 10 GB | 760 GB/s | 320-bit | ✅ + Tensor | ~$350-400 | Q4_K_M 14B | Faster than 2080 Ti but 1GB less VRAM. |
|
||||
| RTX 3080 12GB | 12 GB | 912 GB/s | 384-bit | ✅ + Tensor | ~$350-400 | Q4_K_M 14B+ | True upgrade from 2080 Ti. 48% faster. |
|
||||
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s | **128-bit** ❌ | ✅ + Tensor | $400+ | Q4_K_M 14B | **TRAP.** Slower than $80 GTX 1080. |
|
||||
| RTX 5060 Ti 16GB | 16 GB | 448 GB/s | **128-bit** ❌ | ✅ + Tensor | $450+ | Q4_K_M 14B | **TRAP.** 38% slower than 2080 Ti. |
|
||||
| RTX 4070 Ti Super | 16 GB | 672 GB/s | 256-bit | ✅ + Tensor | $600+ | Q4_K_M 20B | First good 16GB NVIDIA card. |
|
||||
| RTX 3090 | 24 GB | 936 GB/s | 384-bit | ✅ + Tensor | $600-700 | Q4_K_M 32B | Real upgrade. Runs 32B comfortably. |
|
||||
| RX 6650 XT | 8 GB | — | — | ❌ Vulkan only | ~$150 | — | Avoid for LLMs. |
|
||||
| RX 7700 XT | 12 GB | 432 GB/s | — | ❌ Vulkan only | ~$350 | — | Avoid for LLMs. 2× 8-pin, 245W. |
|
||||
|
||||
## Key Specs That Matter for llama.cpp
|
||||
|
||||
- **VRAM**: The model must fit. For reference:
|
||||
- Q4_K_M 7B ≈ 4.7 GB, Q5_K_M ≈ 5.5 GB, Q8_0 ≈ 8 GB
|
||||
- Q4_K_M 14B ≈ 9 GB, Q3_K_M ≈ 6.5 GB, IQ3_M ≈ 7.5 GB
|
||||
- Q3_K_M 20B ≈ 9 GB, IQ2_S ≈ 6 GB
|
||||
- Q4_K_M 32B ≈ 18 GB, IQ2_S ≈ 8.5 GB
|
||||
- Add ~1-2 GB for KV cache (context).
|
||||
- **Two models at once**: Q4_K_M 7B (4.5 GB) + Q3_K_M 14B (6.5 GB) — barely fits 11GB.
|
||||
- **Memory bandwidth**: #1 bottleneck. Every token reads the entire model from VRAM. **Bandwidth directly determines tokens/sec.** All else equal, a card with 50% more bandwidth generates tokens 50% faster for the same model.
|
||||
- **Bus width**: A proxy for bandwidth. 128-bit cards universally bottleneck LLM workloads regardless of VRAM. 256-bit is the minimum for decent inference. 352-bit+ is where things get good.
|
||||
- **CUDA compute capability**: 6.1+ needed for CUDA backend. 7.0+ (Volta and newer) gets full optimization. 7.5+ (Turing) gets tensor core acceleration.
|
||||
- **Tensor cores**: Accelerate FP16 matrix ops. Measurable but modest speedup for llama.cpp CUDA backend.
|
||||
|
||||
## Power Cost Analysis
|
||||
|
||||
At typical US residential rates (~$0.13/kWh), GPU electricity cost is negligible. The idle draw difference between cards (15-30W) is a few cents per day. Even heavy inference (8 hours/day at full TDP) costs $3-13/month total for the entire card. The upfront purchase price dominates — not the power bill.
|
||||
|
||||
Rule of thumb: $80 GPU costs ~$2-3/month to run idle. $600 GPU costs ~$3-5/month idle. The difference is pocket change.
|
||||
|
||||
## nvidia-smi Power Limiting (Reduce Noise/Heat)
|
||||
|
||||
LLM inference is **memory-bandwidth-bound**, not core-clock-bound. You can throttle power with near-zero token speed loss:
|
||||
|
||||
```bash
|
||||
# Check current power limit
|
||||
nvidia-smi -q -d POWER | grep "Power Limit"
|
||||
|
||||
# Cap at 80% of max (keeps 95%+ token speed)
|
||||
sudo nvidia-smi -pl 280 # 3090: 350W → 280W
|
||||
sudo nvidia-smi -pl 200 # 2080 Ti: 250W → 200W
|
||||
|
||||
# Make persistent across reboots
|
||||
sudo nvidia-smi -pm 1 # enable persistence mode
|
||||
sudo nvidia-smi -pl 280 # set power limit
|
||||
```
|
||||
|
||||
Blower cards (Turbo models) benefit most — lowering power from 350W to 280W drops fan RPM significantly because the small impeller screams above 60% speed. Triple-fan open-air cards benefit less since they're already quieter.
|
||||
|
||||
**Impact**: ~20% less power draw for ~3-5% slower tokens. On a 3090 running llama-server 24/7 in a living space, this is the difference between annoying and inaudible.
|
||||
|
||||
## Case Compatibility: Blower vs Open-Air
|
||||
|
||||
For prebuilt/server cases with limited airflow (HP Omen, Dell XPS, SFF builds), **blower-style cards are often better than open-air** despite being louder at stock power:
|
||||
|
||||
| Design | Heat exhaust | Best for | Noise |
|
||||
|---|---|---|---|
|
||||
| **Blower** (Turbo/Founders) | Out the back | Small cases, servers, prebuilts | Louder at full power |
|
||||
| **Open-air** (FTW3/Gaming OC) | Into the case | Full towers, gaming cases | Quieter with good airflow |
|
||||
|
||||
**Why this matters**: An open-air 3090 dumps 350W into a cramped OMEN 30L — the CPU, VRMs, and drives all cook. The blower 3090 Turbo (267mm, 2-slot, 2×8-pin) fits the same case, exhausts heat out the rear, and runs quieter at a 280W power limit than at stock. The blower is also shorter (267mm vs 300mm FTW3) and needs fewer power cables (2×8-pin vs 3×8-pin).
|
||||
|
||||
**Before buying**: always check the case GPU length limit (`dmidecode -t chassis` or physical measurement) plus PSU connector count. Prebuilt PSUs often have fewer PCIe cables than aftermarket units.
|
||||
|
||||
## Server Power Measurement
|
||||
|
||||
For exact costs, a Kill-A-Watt meter (~$15) plugged between the wall and server gives real numbers. Software estimates vary — a 3090 system idling at "100W" from `nvidia-smi` might draw 130W at the wall after PSU inefficiency.
|
||||
|
||||
## System Profiling Commands
|
||||
|
||||
```bash
|
||||
# GPU details
|
||||
nvidia-smi --query-gpu=index,name,memory.total,power.limit --format=csv
|
||||
sudo dmidecode -t baseboard | grep -E "Manufacturer:|Product Name:" # motherboard
|
||||
sudo dmidecode -t system | grep -E "Manufacturer:|Product Name:" # system model
|
||||
sudo dmidecode --type memory | grep -E "Type:|Speed:|Size:" # RAM
|
||||
lscpu | grep "Model name" # CPU
|
||||
|
||||
# Check what's using VRAM
|
||||
nvidia-smi
|
||||
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
|
||||
```
|
||||
|
||||
## Recommendation Heuristic
|
||||
|
||||
1. **$80-100**: GTX 1080 8GB (320 GB/s) — best budget LLM card, CUDA native, faster than any 128-bit modern card
|
||||
2. **$140-170**: GTX 1080 Ti 11GB (484 GB/s) — sweet spot, runs Q4_K_M 14B
|
||||
3. **$250-300**: RTX 2080 Ti 11GB (616 GB/s) — best sub-$300 pick, tensor cores, near-3090 speed per dollar
|
||||
4. **$350-400**: RTX 3080 12GB (912 GB/s) — major speed jump, 384-bit bus
|
||||
5. **$600-700**: RTX 3090 24GB (936 GB/s) — runs 32B models, "buy once done"
|
||||
6. **Skip entirely**: Any 128-bit NVIDIA card (4060 Ti, 5060 Ti) regardless of VRAM. You're paying for VRAM you can't feed fast enough.
|
||||
7. **Skip entirely**: AMD consumer cards for LLM-first setups. Vulkan-only is not worth the tradeoffs.
|
||||
8. **If you also game**: NVIDIA still wins (CUDA + DLSS for gaming, CUDA for LLMs).
|
||||
@@ -0,0 +1,77 @@
|
||||
# Hermes Local Model Configuration (Ollama)
|
||||
|
||||
When using local models via Ollama with Hermes Agent, the main model and each auxiliary task must be configured separately. This reference covers the full wiring.
|
||||
|
||||
## Quick Check
|
||||
|
||||
```bash
|
||||
grep -A3 'provider:' ~/.hermes/config.yaml | grep -B1 'provider:'
|
||||
```
|
||||
|
||||
This shows every provider setting. Any still pointing at a cloud provider that should be local needs updating.
|
||||
|
||||
## Main Model
|
||||
|
||||
```bash
|
||||
hermes config set model.provider ollama
|
||||
hermes config set model.default qwen2.5:14b
|
||||
hermes config set model.base_url http://localhost:11434
|
||||
```
|
||||
|
||||
## Auxiliary Sections
|
||||
|
||||
Each uses a model and must be configured independently. Changing `model.provider` does NOT affect them.
|
||||
|
||||
| Section | Purpose | Priority |
|
||||
|---------|---------|----------|
|
||||
| `auxiliary.vision` | Image analysis | High |
|
||||
| `auxiliary.delegation` | Subagent chat model | High |
|
||||
| `auxiliary.web_extract` | Web page summarization | Medium |
|
||||
| `auxiliary.compression` | Context window compression | Medium |
|
||||
| `auxiliary.approval` | Smart command approval | Low |
|
||||
| `auxiliary.mcp` | MCP server execution | Low |
|
||||
| `auxiliary.curator` | Skill lifecycle | Low |
|
||||
| `auxiliary.title_generation` | Session naming | Low |
|
||||
| `auxiliary.skills_hub` | Skills catalog search | Low |
|
||||
| `auxiliary.kanban_decomposer` | Kanban task breakdown | Low |
|
||||
| `auxiliary.profile_describer` | Profile descriptions | Low |
|
||||
| `auxiliary.triage_specifier` | Model request routing | Low |
|
||||
|
||||
### Per-section config pattern
|
||||
|
||||
```bash
|
||||
hermes config set auxiliary.vision.provider ollama
|
||||
hermes config set auxiliary.vision.model llava:13b
|
||||
hermes config set auxiliary.vision.base_url http://localhost:11434
|
||||
|
||||
hermes config set delegation.provider ollama
|
||||
hermes config set delegation.model qwen2.5:14b
|
||||
hermes config set delegation.base_url http://localhost:11434
|
||||
```
|
||||
|
||||
Repeat for each auxiliary section as needed. Low-priority ones can safely stay on the default cloud provider for speed.
|
||||
|
||||
## Switching Back to Cloud
|
||||
|
||||
```bash
|
||||
hermes config set model.provider deepseek
|
||||
hermes config set model.default deepseek-v4-pro
|
||||
```
|
||||
|
||||
## Gateway Restart
|
||||
|
||||
Config changes require a restart:
|
||||
|
||||
```bash
|
||||
hermes gateway restart
|
||||
```
|
||||
|
||||
**Pitfall:** `hermes gateway restart` is blocked inside the gateway process to prevent restart loops. The user must run it from their own shell terminal.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- Auxiliary models not switched — changing main model does not touch vision/compression/delegation.
|
||||
- VRAM exhaustion — loading vision model while LLM is loaded can OOM. Ollama unloads idle models after 5 min.
|
||||
- Speed expectations — local 14B is slower than cloud API. First cold-load takes 10-30 seconds.
|
||||
- Config changes need `/reset` or relaunch — they don't hot-reload mid-session.
|
||||
- Small cloud models may outpace local ones for latency-sensitive tasks (web extraction, compression).
|
||||
@@ -0,0 +1,168 @@
|
||||
# Hugging Face URL Workflows for llama.cpp
|
||||
|
||||
Use URL-only workflows first. Do not require `hf` or API clients just to find GGUF files, choose a quant, or build a `llama-server` command.
|
||||
|
||||
## Core URLs
|
||||
|
||||
```text
|
||||
Search:
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
|
||||
Search with text:
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
||||
|
||||
Search with size bounds:
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
|
||||
Repo local-app view:
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
|
||||
Repo tree API:
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
|
||||
Repo file tree:
|
||||
https://huggingface.co/<repo>/tree/main
|
||||
```
|
||||
|
||||
## 1. Search for llama.cpp-compatible models
|
||||
|
||||
Start from the models page with `apps=llama.cpp`.
|
||||
|
||||
Use:
|
||||
|
||||
- `search=<term>` for model family names such as `Qwen`, `Gemma`, `Phi`, or `Mistral`
|
||||
- `num_parameters=min:0,max:24B` or similar if the user has hardware limits
|
||||
- `sort=trending` when the user wants popular repos right now
|
||||
|
||||
Do not start with random GGUF repos if the user has not chosen a model family yet. Search first, shortlist second.
|
||||
|
||||
Example: https://huggingface.co/models?search=Qwen&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
|
||||
## 2. Use the local-app page for the recommended quant
|
||||
|
||||
Open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
```
|
||||
|
||||
Extract, in order:
|
||||
|
||||
1. The exact `Use this model` snippet, if it is visible as text
|
||||
2. The `Hardware compatibility` section from the fetched page text or HTML:
|
||||
- quant label
|
||||
- file size
|
||||
- bit-depth grouping
|
||||
3. Any extra launch flags shown in the snippet, such as `--jinja`
|
||||
|
||||
Treat the HF local-app snippet as the source of truth when it is visible.
|
||||
|
||||
Do this by reading the URL itself, not by assuming the UI rendered in a browser. If the fetched page source does not expose `Hardware compatibility`, say that the section was not text-visible and fall back to the tree API plus generic guidance from `quantization.md`.
|
||||
|
||||
## 3. Confirm exact files from the tree API
|
||||
|
||||
Open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Treat the JSON response as the source of truth for repo inventory.
|
||||
|
||||
Keep entries where:
|
||||
|
||||
- `type` is `file`
|
||||
- `path` ends with `.gguf`
|
||||
|
||||
Use these fields:
|
||||
|
||||
- `path` for the filename and subdirectory
|
||||
- `size` for the byte size
|
||||
- optionally `lfs.size` to confirm the LFS payload size
|
||||
|
||||
Separate files into:
|
||||
|
||||
- quantized single-file checkpoints, for example `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
||||
- projector weights, usually `mmproj-*.gguf`
|
||||
- BF16 shard files, usually under `BF16/`
|
||||
- everything else
|
||||
|
||||
Ignore unless the user asks:
|
||||
|
||||
- `README.md`
|
||||
- imatrix or calibration blobs
|
||||
|
||||
Use `https://huggingface.co/<repo>/tree/main` only as a human fallback if the API endpoint fails or the user wants the web view.
|
||||
|
||||
## 4. Build the command
|
||||
|
||||
Preferred order:
|
||||
|
||||
1. Copy the exact HF snippet from the local-app page
|
||||
2. If the page gives a clean quant label, use shorthand selection:
|
||||
|
||||
```bash
|
||||
llama-server -hf <repo>:<QUANT>
|
||||
```
|
||||
|
||||
3. If you need an exact file from the tree API, use the file-specific form:
|
||||
|
||||
```bash
|
||||
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
||||
```
|
||||
|
||||
4. For CLI usage instead of a server, use:
|
||||
|
||||
```bash
|
||||
llama-cli -hf <repo>:<QUANT>
|
||||
```
|
||||
|
||||
Use the exact-file form when the repo uses custom labels or nonstandard naming that could make `:<QUANT>` ambiguous.
|
||||
|
||||
## 5. Example: `unsloth/Qwen3.6-35B-A3B-GGUF`
|
||||
|
||||
Use these URLs:
|
||||
|
||||
```text
|
||||
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
|
||||
https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
|
||||
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main
|
||||
```
|
||||
|
||||
On the local-app page, the hardware compatibility section can expose entries such as:
|
||||
|
||||
- `UD-IQ4_XS` - 17.7 GB
|
||||
- `UD-Q4_K_S` - 20.9 GB
|
||||
- `UD-Q4_K_M` - 22.1 GB
|
||||
- `UD-Q5_K_M` - 26.5 GB
|
||||
- `UD-Q6_K` - 29.3 GB
|
||||
- `Q8_0` - 36.9 GB
|
||||
|
||||
On the tree API, you can confirm exact filenames such as:
|
||||
|
||||
- `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf`
|
||||
- `Qwen3.6-35B-A3B-UD-Q5_K_M.gguf`
|
||||
- `Qwen3.6-35B-A3B-UD-Q6_K.gguf`
|
||||
- `Qwen3.6-35B-A3B-Q8_0.gguf`
|
||||
- `mmproj-F16.gguf`
|
||||
|
||||
Good final output for this repo:
|
||||
|
||||
```text
|
||||
Repo: unsloth/Qwen3.6-35B-A3B-GGUF
|
||||
Recommended quant from HF: UD-Q4_K_M (22.1 GB)
|
||||
llama-server: llama-server --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf
|
||||
Other GGUFs:
|
||||
- Qwen3.6-35B-A3B-UD-Q5_K_M.gguf - 26.5 GB
|
||||
- Qwen3.6-35B-A3B-UD-Q6_K.gguf - 29.3 GB
|
||||
- Qwen3.6-35B-A3B-Q8_0.gguf - 36.9 GB
|
||||
Projector:
|
||||
- mmproj-F16.gguf - 899 MB
|
||||
```
|
||||
|
||||
## Notes
|
||||
|
||||
- Repo-specific quant labels matter. Do not rewrite `UD-Q4_K_M` to `Q4_K_M` unless the page itself does.
|
||||
- `mmproj` files are projector weights for multimodal models, not the main language model checkpoint.
|
||||
- If the HF hardware compatibility panel is missing because the user has no hardware profile configured, or because the fetched page source did not expose it, still use the tree API plus generic quant guidance from `quantization.md`.
|
||||
- If the repo already has GGUFs, do not jump straight to conversion workflows.
|
||||
@@ -0,0 +1,127 @@
|
||||
# Migrating from External AI APIs to llama-server
|
||||
|
||||
Pattern: replace external LLM API calls (Gemini, OpenAI, Anthropic) with a local llama-server behind nginx.
|
||||
|
||||
## When to use
|
||||
|
||||
- Broken/missing API keys in a legacy codebase
|
||||
- Want to eliminate external API dependency for privacy, cost, or reliability
|
||||
- Already have llama-server running locally (see `references/deployment-patterns.md`)
|
||||
|
||||
## Discovery: find all AI calls
|
||||
|
||||
```bash
|
||||
# Find every fetch() or HTTP call in the project
|
||||
grep -rn 'fetch(' --include='*.js' --include='*.html' .
|
||||
# Focus on external AI APIs
|
||||
grep -rn 'googleapis\|openai.com\|api.anthropic' --include='*.js' .
|
||||
```
|
||||
|
||||
Real-world example from an auto-repair shop web app — 3 dead Gemini calls and 1 working local LLM call already in the codebase.
|
||||
|
||||
## Conversion recipe
|
||||
|
||||
### Gemini → llama.cpp
|
||||
|
||||
**Before (Gemini):**
|
||||
```javascript
|
||||
const apiKey = "AIzaSy...nZqc";
|
||||
const apiUrl = `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=${apiKey}`;
|
||||
|
||||
const payload = {
|
||||
contents: [{
|
||||
role: "user",
|
||||
parts: [{ text: prompt }]
|
||||
}]
|
||||
};
|
||||
|
||||
const response = await fetch(apiUrl, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
|
||||
const result = await response.json();
|
||||
const text = result.candidates[0].content.parts[0].text;
|
||||
```
|
||||
|
||||
**After (llama-server):**
|
||||
```javascript
|
||||
const response = await fetch('/llm/v1/chat/completions', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
messages: [
|
||||
{role:'system', content: prompt}, // instructions
|
||||
{role:'user', content: userData} // actual data
|
||||
],
|
||||
temperature: 0,
|
||||
max_tokens: 500
|
||||
})
|
||||
});
|
||||
|
||||
const result = await response.json();
|
||||
const text = result.choices[0].message.content;
|
||||
```
|
||||
|
||||
### Response path mapping
|
||||
|
||||
| Provider | Response text path |
|
||||
|---|---|
|
||||
| Gemini | `result.candidates[0].content.parts[0].text` |
|
||||
| OpenAI | `result.choices[0].message.content` |
|
||||
| llama.cpp (`/v1/chat/completions`) | `result.choices[0].message.content` |
|
||||
|
||||
### Payload format mapping
|
||||
|
||||
| Provider | Format |
|
||||
|---|---|
|
||||
| Gemini | `{contents: [{role: "user", parts: [{text}]}]}` |
|
||||
| OpenAI / llama.cpp | `{messages: [{role: "user", content: text}]}` |
|
||||
|
||||
## System prompt from user prompt
|
||||
|
||||
Gemini calls often have the full instruction text as a single `user`-role message. For llama.cpp, split it:
|
||||
|
||||
- **System message** = the instructions/formatting rules (the "you are an expert" part)
|
||||
- **User message** = the actual data to process (service names, OCR text, etc.)
|
||||
|
||||
This mirrors the already-working pattern in the scan screenshot feature.
|
||||
|
||||
## Testing after migration
|
||||
|
||||
Always test end-to-end with the actual prompt the code uses:
|
||||
|
||||
```bash
|
||||
# Test through nginx (what the browser sees)
|
||||
curl -sk https://localhost:3447/llm/v1/chat/completions \
|
||||
-X POST -H 'Content-Type: application/json' \
|
||||
-d '{"messages":[{"role":"system","content":"Your prompt here"},{"role":"user","content":"Input data"}],"temperature":0,"max_tokens":300}'
|
||||
```
|
||||
|
||||
Checklist:
|
||||
- [ ] Response parses correctly (JSON keys match, LEVEL/EXPLANATION format intact)
|
||||
- [ ] Error handling still works (network error, empty response)
|
||||
- [ ] Existing fallbacks still trigger on failure
|
||||
|
||||
## Model quality expectations
|
||||
|
||||
1.5B models (Qwen2.5-1.5B) handle extraction and classification reasonably but:
|
||||
- Priority/severity judgment is weaker than larger models
|
||||
- May misclassify edge cases (2mm brake pads as "RECOMMENDED" not "CRITICAL")
|
||||
- Solution: keep keyword-based fallback rules that override LLM judgments for known dangerous conditions
|
||||
|
||||
## Nginx prerequisite
|
||||
|
||||
The frontend uses a relative URL: `fetch('/llm/v1/chat/completions')`. This works because nginx serves both the static site and proxies `/llm/` to llama-server on the same port:
|
||||
|
||||
```nginx
|
||||
location /llm/ {
|
||||
proxy_pass http://127.0.0.1:8081/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host $host;
|
||||
proxy_read_timeout 120s;
|
||||
}
|
||||
```
|
||||
|
||||
No CORS, no separate port, no API key management — the browser thinks it's calling its own server.
|
||||
@@ -0,0 +1,176 @@
|
||||
# Ollama Deployment (Alternative to standalone llama-server)
|
||||
|
||||
Ollama provides on-demand model loading with automatic 5-minute idle unload.
|
||||
Unlike standalone llama-server which keeps a model resident in VRAM permanently,
|
||||
Ollama loads on first request and frees VRAM when idle — ideal for shared GPU
|
||||
scenarios (work + inference on the same card).
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
```
|
||||
|
||||
Creates systemd service `ollama`, listens on `127.0.0.1:11434`.
|
||||
|
||||
## Pull a model
|
||||
|
||||
```bash
|
||||
# Text inference
|
||||
ollama pull qwen2.5:7b
|
||||
|
||||
# Vision (image understanding)
|
||||
ollama pull llava:7b
|
||||
```
|
||||
|
||||
Models auto-unload after 5 minutes idle. `ollama ps` shows currently loaded models.
|
||||
|
||||
## Nginx reverse proxy — ⚠️ CRITICAL: Host header
|
||||
|
||||
**Ollama rejects requests where the `Host` header is not `localhost`.** When
|
||||
proxying through nginx, you MUST override the Host header:
|
||||
|
||||
```nginx
|
||||
location /llm/ {
|
||||
proxy_pass http://127.0.0.1:11434/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host "localhost"; # ← REQUIRED — Ollama rejects $host
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 120s; # allow for cold-start model load
|
||||
}
|
||||
```
|
||||
|
||||
Without this, nginx forwards the original Host header (`grajmedia.duckdns.org`,
|
||||
`192.168.50.98`, etc.) and Ollama returns **403 Forbidden** immediately
|
||||
(microsecond reject — not a model-loading delay).
|
||||
|
||||
Same fix applies for vision proxy:
|
||||
|
||||
```nginx
|
||||
location /vision/ {
|
||||
proxy_pass http://127.0.0.1:11434/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Host "localhost";
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_buffering off;
|
||||
proxy_read_timeout 300s; # vision models load slower
|
||||
}
|
||||
```
|
||||
|
||||
## OpenAI-compatible endpoint
|
||||
|
||||
Ollama serves `/v1/chat/completions` with the same response format as
|
||||
llama-server and the OpenAI API:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "chatcmpl-898",
|
||||
"object": "chat.completion",
|
||||
"created": 1781059955,
|
||||
"model": "qwen2.5:7b",
|
||||
"choices": [{
|
||||
"index": 0,
|
||||
"message": { "role": "assistant", "content": "Hi!" },
|
||||
"finish_reason": "stop"
|
||||
}],
|
||||
"usage": { "prompt_tokens": 34, "completion_tokens": 3, "total_tokens": 37 }
|
||||
}
|
||||
```
|
||||
|
||||
## Migrating from llama-server to Ollama
|
||||
|
||||
llama-server does not require a `model` field in the request body. Ollama does.
|
||||
When migrating, add the model name to every API call:
|
||||
|
||||
**Before (llama-server):**
|
||||
```javascript
|
||||
body: JSON.stringify({
|
||||
messages: [...],
|
||||
temperature: 0,
|
||||
max_tokens: 500
|
||||
})
|
||||
```
|
||||
|
||||
**After (Ollama):**
|
||||
```javascript
|
||||
body: JSON.stringify({
|
||||
model: 'qwen2.5:7b',
|
||||
messages: [...],
|
||||
temperature: 0,
|
||||
max_tokens: 500
|
||||
})
|
||||
```
|
||||
|
||||
Then update the nginx proxy target and restart:
|
||||
|
||||
```bash
|
||||
# Change proxy_pass in nginx config, then:
|
||||
sudo nginx -t && sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
Stop the old llama-server to free VRAM:
|
||||
|
||||
```bash
|
||||
kill <llama-server-pid>
|
||||
```
|
||||
|
||||
## Vision model API
|
||||
|
||||
Ollama's vision API uses the native chat format with an `images` array.
|
||||
Base64 prefix `data:image/png;base64,` is NOT included — just the raw base64.
|
||||
|
||||
```javascript
|
||||
const b64 = await new Promise(resolve => {
|
||||
const r = new FileReader();
|
||||
r.onload = () => resolve(r.result.split(',')[1]); // strip data:image/...;base64,
|
||||
r.readAsDataURL(file);
|
||||
});
|
||||
|
||||
const resp = await fetch('/vision/api/chat', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'llava:7b',
|
||||
messages: [{
|
||||
role: 'user',
|
||||
content: 'Your system prompt + instructions + schema',
|
||||
images: [b64]
|
||||
}],
|
||||
stream: false
|
||||
})
|
||||
});
|
||||
const data = await resp.json();
|
||||
const content = data.message.content; // note: not choices[0].message (that's /v1/chat/completions format)
|
||||
```
|
||||
|
||||
**Key difference:** The native `/api/chat` endpoint returns `data.message.content`.
|
||||
The OpenAI-compatible `/v1/chat/completions` returns `data.choices[0].message.content`.
|
||||
Use `/v1/chat/completions` for text models and `/api/chat` for vision if you use
|
||||
the native format — but send the `model` field either way.
|
||||
|
||||
## client_max_body_size
|
||||
|
||||
Vision requests include a base64 image which can be 2-5 MB (or more for large
|
||||
screenshots). nginx default `client_max_body_size` is 1 MB — requests over this
|
||||
are silently rejected with 413. Set in the server block:
|
||||
|
||||
```nginx
|
||||
client_max_body_size 50m;
|
||||
```
|
||||
|
||||
## Replacing Tesseract OCR with a vision model
|
||||
|
||||
For complex table layouts (schedules, invoices, spreadsheets), Tesseract cannot
|
||||
reliably preserve row/column structure. A VLM (vision-language model) like
|
||||
llava:7b sees the layout visually and outputs structured JSON directly,
|
||||
eliminating the need for:
|
||||
- Image preprocessing (upscaling, sharpening, contrast)
|
||||
- TSV bounding-box parsing
|
||||
- Coordinate-based row clustering
|
||||
- OCR text → LLM text → JSON pipeline
|
||||
|
||||
**Old pipeline:** Screenshot → preprocessing → Tesseract → TSV bbox clustering →
|
||||
text LLM → JSON
|
||||
|
||||
**New pipeline:** Screenshot → base64 → VLM → JSON
|
||||
@@ -0,0 +1,89 @@
|
||||
# Performance Optimization Guide
|
||||
|
||||
Maximize llama.cpp inference speed and efficiency.
|
||||
|
||||
## CPU Optimization
|
||||
|
||||
### Thread tuning
|
||||
```bash
|
||||
# Set threads (default: physical cores)
|
||||
./llama-cli -m model.gguf -t 8
|
||||
|
||||
# For AMD Ryzen 9 7950X (16 cores, 32 threads)
|
||||
-t 16 # Best: physical cores
|
||||
|
||||
# Avoid hyperthreading (slower for matrix ops)
|
||||
```
|
||||
|
||||
### BLAS acceleration
|
||||
```bash
|
||||
# OpenBLAS (faster matrix ops)
|
||||
make LLAMA_OPENBLAS=1
|
||||
|
||||
# BLAS gives 2-3× speedup
|
||||
```
|
||||
|
||||
## GPU Offloading
|
||||
|
||||
### Layer offloading
|
||||
```bash
|
||||
# Offload 35 layers to GPU (hybrid mode)
|
||||
./llama-cli -m model.gguf -ngl 35
|
||||
|
||||
# Offload all layers
|
||||
./llama-cli -m model.gguf -ngl 999
|
||||
|
||||
# Find optimal value:
|
||||
# Start with -ngl 999
|
||||
# If OOM, reduce by 5 until fits
|
||||
```
|
||||
|
||||
### Memory usage
|
||||
```bash
|
||||
# Check VRAM usage
|
||||
nvidia-smi dmon
|
||||
|
||||
# Reduce context if needed
|
||||
./llama-cli -m model.gguf -c 2048 # 2K context instead of 4K
|
||||
```
|
||||
|
||||
## Batch Processing
|
||||
|
||||
```bash
|
||||
# Increase batch size for throughput
|
||||
./llama-cli -m model.gguf -b 512 # Default: 512
|
||||
|
||||
# Physical batch (GPU)
|
||||
--ubatch 128 # Process 128 tokens at once
|
||||
```
|
||||
|
||||
## Context Management
|
||||
|
||||
```bash
|
||||
# Default context (512 tokens)
|
||||
-c 512
|
||||
|
||||
# Longer context (slower, more memory)
|
||||
-c 4096
|
||||
|
||||
# Very long context (if model supports)
|
||||
-c 32768
|
||||
```
|
||||
|
||||
## Benchmarks
|
||||
|
||||
### CPU Performance (Llama 2-7B Q4_K_M)
|
||||
|
||||
| Setup | Speed | Notes |
|
||||
|-------|-------|-------|
|
||||
| Apple M3 Max | 50 tok/s | Metal acceleration |
|
||||
| AMD 7950X (16c) | 35 tok/s | OpenBLAS |
|
||||
| Intel i9-13900K | 30 tok/s | AVX2 |
|
||||
|
||||
### GPU Offloading (RTX 4090)
|
||||
|
||||
| Layers GPU | Speed | VRAM |
|
||||
|------------|-------|------|
|
||||
| 0 (CPU only) | 30 tok/s | 0 GB |
|
||||
| 20 (hybrid) | 80 tok/s | 8 GB |
|
||||
| 35 (all) | 120 tok/s | 12 GB |
|
||||
@@ -0,0 +1,243 @@
|
||||
# GGUF Quantization Guide
|
||||
|
||||
Complete guide to GGUF quantization formats and model conversion.
|
||||
|
||||
## Hub-first quant selection
|
||||
|
||||
Before using generic tables, open the model repo with:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
```
|
||||
|
||||
Prefer the exact quant labels and sizes shown in the `Hardware compatibility` section of the fetched `?local-app=llama.cpp` page text or HTML. Then confirm the matching filenames in:
|
||||
|
||||
```text
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Use the Hub page first, and only fall back to the generic heuristics below when the repo page does not expose a clear recommendation.
|
||||
|
||||
## Quantization Overview
|
||||
|
||||
**GGUF** (GPT-Generated Unified Format) - Standard format for llama.cpp models.
|
||||
|
||||
### Format Comparison
|
||||
|
||||
| Format | Perplexity | Size (7B) | Tokens/sec | Notes |
|
||||
|--------|------------|-----------|------------|-------|
|
||||
| FP16 | 5.9565 (baseline) | 13.0 GB | 15 tok/s | Original quality |
|
||||
| Q8_0 | 5.9584 (+0.03%) | 7.0 GB | 25 tok/s | Nearly lossless |
|
||||
| **Q6_K** | 5.9642 (+0.13%) | 5.5 GB | 30 tok/s | Best quality/size |
|
||||
| **Q5_K_M** | 5.9796 (+0.39%) | 4.8 GB | 35 tok/s | Balanced |
|
||||
| **Q4_K_M** | 6.0565 (+1.68%) | 4.1 GB | 40 tok/s | **Recommended** |
|
||||
| Q4_K_S | 6.1125 (+2.62%) | 3.9 GB | 42 tok/s | Faster, lower quality |
|
||||
| Q3_K_M | 6.3184 (+6.07%) | 3.3 GB | 45 tok/s | Small models only |
|
||||
| Q2_K | 6.8673 (+15.3%) | 2.7 GB | 50 tok/s | Not recommended |
|
||||
|
||||
**Recommendation**: Use **Q4_K_M** for best balance of quality and speed.
|
||||
|
||||
## Converting Models
|
||||
|
||||
### Hugging Face to GGUF
|
||||
|
||||
```bash
|
||||
# 1. Download Hugging Face model
|
||||
hf download meta-llama/Llama-2-7b-chat-hf \
|
||||
--local-dir models/llama-2-7b-chat/
|
||||
|
||||
# 2. Convert to FP16 GGUF
|
||||
python convert_hf_to_gguf.py \
|
||||
models/llama-2-7b-chat/ \
|
||||
--outtype f16 \
|
||||
--outfile models/llama-2-7b-chat-f16.gguf
|
||||
|
||||
# 3. Quantize to Q4_K_M
|
||||
./llama-quantize \
|
||||
models/llama-2-7b-chat-f16.gguf \
|
||||
models/llama-2-7b-chat-Q4_K_M.gguf \
|
||||
Q4_K_M
|
||||
```
|
||||
|
||||
### Batch quantization
|
||||
|
||||
```bash
|
||||
# Quantize to multiple formats
|
||||
for quant in Q4_K_M Q5_K_M Q6_K Q8_0; do
|
||||
./llama-quantize \
|
||||
model-f16.gguf \
|
||||
model-${quant}.gguf \
|
||||
$quant
|
||||
done
|
||||
```
|
||||
|
||||
## K-Quantization Methods
|
||||
|
||||
**K-quants** use mixed precision for better quality:
|
||||
- Attention weights: Higher precision
|
||||
- Feed-forward weights: Lower precision
|
||||
|
||||
**Variants**:
|
||||
- `_S` (Small): Faster, lower quality
|
||||
- `_M` (Medium): Balanced (recommended)
|
||||
- `_L` (Large): Better quality, larger size
|
||||
|
||||
**Example**: `Q4_K_M`
|
||||
- `Q4`: 4-bit quantization
|
||||
- `K`: Mixed precision method
|
||||
- `M`: Medium quality
|
||||
|
||||
## Quality Testing
|
||||
|
||||
```bash
|
||||
# Calculate perplexity (quality metric)
|
||||
./llama-perplexity \
|
||||
-m model.gguf \
|
||||
-f wikitext-2-raw/wiki.test.raw \
|
||||
-c 512
|
||||
|
||||
# Lower perplexity = better quality
|
||||
# Baseline (FP16): ~5.96
|
||||
# Q4_K_M: ~6.06 (+1.7%)
|
||||
# Q2_K: ~6.87 (+15.3% - too much degradation)
|
||||
```
|
||||
|
||||
## Use Case Guide
|
||||
|
||||
### General purpose (chatbots, assistants)
|
||||
```
|
||||
Q4_K_M - Best balance
|
||||
Q5_K_M - If you have extra RAM
|
||||
```
|
||||
|
||||
### Code generation
|
||||
```
|
||||
Q5_K_M or Q6_K - Higher precision helps with code
|
||||
```
|
||||
|
||||
### Creative writing
|
||||
```
|
||||
Q4_K_M - Sufficient quality
|
||||
Q3_K_M - Acceptable for draft generation
|
||||
```
|
||||
|
||||
### Technical/medical
|
||||
```
|
||||
Q6_K or Q8_0 - Maximum accuracy
|
||||
```
|
||||
|
||||
### Edge devices (Raspberry Pi)
|
||||
```
|
||||
Q2_K or Q3_K_S - Fit in limited RAM
|
||||
```
|
||||
|
||||
## Model Size Scaling
|
||||
|
||||
### 7B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 2.7 GB | 5 GB |
|
||||
| Q3_K_M | 3.3 GB | 6 GB |
|
||||
| Q4_K_M | 4.1 GB | 7 GB |
|
||||
| Q5_K_M | 4.8 GB | 8 GB |
|
||||
| Q6_K | 5.5 GB | 9 GB |
|
||||
| Q8_0 | 7.0 GB | 11 GB |
|
||||
|
||||
### 13B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 5.1 GB | 8 GB |
|
||||
| Q3_K_M | 6.2 GB | 10 GB |
|
||||
| Q4_K_M | 7.9 GB | 12 GB |
|
||||
| Q5_K_M | 9.2 GB | 14 GB |
|
||||
| Q6_K | 10.7 GB | 16 GB |
|
||||
|
||||
### 70B parameter models
|
||||
|
||||
| Format | Size | RAM needed |
|
||||
|--------|------|------------|
|
||||
| Q2_K | 26 GB | 32 GB |
|
||||
| Q3_K_M | 32 GB | 40 GB |
|
||||
| Q4_K_M | 41 GB | 48 GB |
|
||||
| Q4_K_S | 39 GB | 46 GB |
|
||||
| Q5_K_M | 48 GB | 56 GB |
|
||||
|
||||
**Recommendation for 70B**: Use Q3_K_M or Q4_K_S to fit in consumer hardware.
|
||||
|
||||
## Finding Pre-Quantized Models
|
||||
|
||||
Use the Hub search with the llama.cpp app filter:
|
||||
|
||||
```text
|
||||
https://huggingface.co/models?apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
|
||||
https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
|
||||
```
|
||||
|
||||
For a specific repo, open:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
https://huggingface.co/api/models/<repo>/tree/main?recursive=true
|
||||
```
|
||||
|
||||
Then launch directly from the Hub without extra Hub tooling:
|
||||
|
||||
```bash
|
||||
llama-cli -hf <repo>:Q4_K_M
|
||||
llama-server -hf <repo>:Q4_K_M
|
||||
```
|
||||
|
||||
If you need the exact file name from the tree API:
|
||||
|
||||
```bash
|
||||
llama-server --hf-repo <repo> --hf-file <filename.gguf>
|
||||
```
|
||||
|
||||
## Importance Matrices (imatrix)
|
||||
|
||||
**What**: Calibration data to improve quantization quality.
|
||||
|
||||
**Benefits**:
|
||||
- 10-20% perplexity improvement with Q4
|
||||
- Essential for Q3 and below
|
||||
|
||||
**Usage**:
|
||||
```bash
|
||||
# 1. Generate importance matrix
|
||||
./llama-imatrix \
|
||||
-m model-f16.gguf \
|
||||
-f calibration-data.txt \
|
||||
-o model.imatrix
|
||||
|
||||
# 2. Quantize with imatrix
|
||||
./llama-quantize \
|
||||
--imatrix model.imatrix \
|
||||
model-f16.gguf \
|
||||
model-Q4_K_M.gguf \
|
||||
Q4_K_M
|
||||
```
|
||||
|
||||
**Calibration data**:
|
||||
- Use domain-specific text (e.g., code for code models)
|
||||
- ~100MB of representative text
|
||||
- Higher quality data = better quantization
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Model outputs gibberish**:
|
||||
- Quantization too aggressive (Q2_K)
|
||||
- Try Q4_K_M or Q5_K_M
|
||||
- Verify model converted correctly
|
||||
|
||||
**Out of memory**:
|
||||
- Use lower quantization (Q4_K_S instead of Q5_K_M)
|
||||
- Offload fewer layers to GPU (`-ngl`)
|
||||
- Use smaller context (`-c 2048`)
|
||||
|
||||
**Slow inference**:
|
||||
- Higher quantization uses more compute
|
||||
- Q8_0 much slower than Q4_K_M
|
||||
- Consider speed vs quality trade-off
|
||||
@@ -0,0 +1,150 @@
|
||||
# Server Deployment Guide
|
||||
|
||||
Production deployment of llama.cpp server with OpenAI-compatible API.
|
||||
|
||||
## Direct from Hugging Face Hub
|
||||
|
||||
Prefer the model repo's local-app page first:
|
||||
|
||||
```text
|
||||
https://huggingface.co/<repo>?local-app=llama.cpp
|
||||
```
|
||||
|
||||
If the page shows an exact snippet, copy it. If not, use one of these forms:
|
||||
|
||||
```bash
|
||||
# Choose a quant label directly from the Hub repo
|
||||
llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
|
||||
```
|
||||
|
||||
```bash
|
||||
# Pin an exact GGUF file from the repo tree
|
||||
llama-server \
|
||||
--hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
|
||||
--hf-file Phi-3-mini-4k-instruct-q4.gguf \
|
||||
-c 4096
|
||||
```
|
||||
|
||||
Use the file-specific form when the repo has custom naming or when you already extracted the exact filename from the tree API.
|
||||
|
||||
## Server Modes
|
||||
|
||||
### llama-server
|
||||
|
||||
```bash
|
||||
# Basic server
|
||||
./llama-server \
|
||||
-m models/llama-2-7b-chat.Q4_K_M.gguf \
|
||||
--host 0.0.0.0 \
|
||||
--port 8080 \
|
||||
-c 4096 # Context size
|
||||
|
||||
# With GPU acceleration
|
||||
./llama-server \
|
||||
-m models/llama-2-70b.Q4_K_M.gguf \
|
||||
-ngl 40 # Offload 40 layers to GPU
|
||||
```
|
||||
|
||||
## OpenAI-Compatible API
|
||||
|
||||
### Chat completions
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "llama-2",
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are helpful"},
|
||||
{"role": "user", "content": "Hello"}
|
||||
],
|
||||
"temperature": 0.7,
|
||||
"max_tokens": 100
|
||||
}'
|
||||
```
|
||||
|
||||
### Streaming
|
||||
```bash
|
||||
curl http://localhost:8080/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "llama-2",
|
||||
"messages": [{"role": "user", "content": "Count to 10"}],
|
||||
"stream": true
|
||||
}'
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
**Dockerfile**:
|
||||
```dockerfile
|
||||
FROM ubuntu:22.04
|
||||
RUN apt-get update && apt-get install -y git build-essential
|
||||
RUN git clone https://github.com/ggerganov/llama.cpp
|
||||
WORKDIR /llama.cpp
|
||||
RUN make LLAMA_CUDA=1
|
||||
COPY models/ /models/
|
||||
EXPOSE 8080
|
||||
CMD ["./llama-server", "-m", "/models/model.gguf", "--host", "0.0.0.0", "--port", "8080"]
|
||||
```
|
||||
|
||||
**Run**:
|
||||
```bash
|
||||
docker run --gpus all -p 8080:8080 llama-cpp:latest
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
```bash
|
||||
# Server metrics endpoint
|
||||
curl http://localhost:8080/metrics
|
||||
|
||||
# Health check
|
||||
curl http://localhost:8080/health
|
||||
```
|
||||
|
||||
**Metrics**:
|
||||
- requests_total
|
||||
- tokens_generated
|
||||
- prompt_tokens
|
||||
- completion_tokens
|
||||
- kv_cache_tokens
|
||||
|
||||
## Load Balancing
|
||||
|
||||
**NGINX**:
|
||||
```nginx
|
||||
upstream llama_cpp {
|
||||
server llama1:8080;
|
||||
server llama2:8080;
|
||||
}
|
||||
|
||||
server {
|
||||
location / {
|
||||
proxy_pass http://llama_cpp;
|
||||
proxy_read_timeout 300s;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
**Parallel requests**:
|
||||
```bash
|
||||
./llama-server \
|
||||
-m model.gguf \
|
||||
-np 4 # 4 parallel slots
|
||||
```
|
||||
|
||||
**Continuous batching**:
|
||||
```bash
|
||||
./llama-server \
|
||||
-m model.gguf \
|
||||
--cont-batching # Enable continuous batching
|
||||
```
|
||||
|
||||
**Context caching**:
|
||||
```bash
|
||||
./llama-server \
|
||||
-m model.gguf \
|
||||
--cache-prompt # Cache processed prompts
|
||||
```
|
||||
@@ -0,0 +1,120 @@
|
||||
# Structured Extraction from OCR/Messy Text
|
||||
|
||||
Pattern: use a small local LLM (1-3B params) to extract structured JSON from messy, semi-structured text — OCR output, logs, emails, forms.
|
||||
|
||||
## Model recommendation
|
||||
|
||||
**Qwen2.5-1.5B-Instruct Q4_K_M** (~1 GB) is excellent for this task class. It follows structured output instructions well at this size, runs at ~30 tok/s on CPU-only (4 threads), and loads in under 500ms. The 32K context window is overkill for extraction tasks; `-c 4096` is sufficient.
|
||||
|
||||
Other viable options at similar size: Llama-3.2-3B, Gemma-2-2B.
|
||||
|
||||
## Prompt template
|
||||
|
||||
The key insight: small models need explicit rules about what NOT to include, not just what TO include. The system prompt must call out specific contamination patterns.
|
||||
|
||||
```
|
||||
Extract appointment details from OCR text. Return ONLY valid JSON, no markdown.
|
||||
|
||||
RULES:
|
||||
- customerName: person name before phone number. Strip "RO" prefix. NEVER include advisor names.
|
||||
- customerPhone: 10 digits only, no dashes
|
||||
- customerEmail: actual email with @. If text after phone is advisor+opcode, leave empty.
|
||||
- vin: 17-char uppercase VIN
|
||||
- vehicleInfo: year + make + model
|
||||
- serviceType: work description after opcode
|
||||
- opCode: only bracketed code like [REP] — no advisor names
|
||||
- appointmentTime: 24h format (e.g. "09:00")
|
||||
- duration: integer minutes
|
||||
|
||||
JSON: {"appointments":[{"customerName":"","customerPhone":"","customerEmail":"","vin":"","vehicleInfo":"","serviceType":"","opCode":"","appointmentTime":"","duration":0}]}
|
||||
```
|
||||
|
||||
**Critical rules that fixed real failures**:
|
||||
- `NEVER include advisor names` — without this, the model puts "Rrahman Grajqevci [REP]" in the opCode field
|
||||
- `Strip "RO" prefix` — OCR text often has "RO Gary Bowers"; the model needs explicit instruction to drop RO
|
||||
- `If text after phone is advisor+opcode, leave empty` — prevents email field from catching advisor name + bracket pattern
|
||||
|
||||
## Temperature
|
||||
|
||||
Always use `temperature: 0` for extraction tasks. Any non-zero temperature introduces field hallucination risk.
|
||||
|
||||
## Markdown fence stripping
|
||||
|
||||
Small models reliably wrap JSON output in ``` fences even when told not to. Always strip client-side:
|
||||
|
||||
```javascript
|
||||
raw = raw.replace(/^```(?:json)?\s*\n?/i, '').replace(/\n?```\s*$/i, '');
|
||||
```
|
||||
|
||||
## Field normalization
|
||||
|
||||
Post-extraction normalization avoids subtle bugs:
|
||||
|
||||
```javascript
|
||||
appointments = parsed.appointments.map(a => ({
|
||||
customerName: a.customerName || '',
|
||||
customerPhone: (a.customerPhone || '').replace(/\D/g, ''), // digits only
|
||||
customerEmail: a.customerEmail || '',
|
||||
vin: (a.vin || '').toUpperCase(),
|
||||
vehicleInfo: a.vehicleInfo || '',
|
||||
serviceType: a.serviceType || '',
|
||||
appointmentTime: a.appointmentTime || '',
|
||||
duration: parseInt(a.duration) || 60,
|
||||
notes: a.opCode ? 'RO: ' + a.opCode : ''
|
||||
}));
|
||||
```
|
||||
|
||||
## OCR Preprocessing (Tesseract time-digit failures)
|
||||
|
||||
Small digits in narrow table columns (e.g., appointment times like 9:00, 12:00, 3:00) are frequently misread by Tesseract as identical values. Upscaling + sharpening before OCR significantly improves digit recognition.
|
||||
|
||||
### Browser-side Canvas preprocessing
|
||||
|
||||
Before passing the image to Tesseract.js, preprocess on a canvas:
|
||||
|
||||
```javascript
|
||||
const preprocessed = await new Promise((resolve, reject) => {
|
||||
const img = new Image();
|
||||
img.onload = () => {
|
||||
const canvas = document.createElement('canvas');
|
||||
const ctx = canvas.getContext('2d');
|
||||
// Upscale 2x — critical for small text
|
||||
canvas.width = img.width * 2;
|
||||
canvas.height = img.height * 2;
|
||||
// Boost contrast and brightness
|
||||
ctx.filter = 'contrast(1.2) brightness(1.1)';
|
||||
ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
|
||||
// Unsharp mask: overlay semi-transparent shifted copies
|
||||
ctx.filter = 'none';
|
||||
ctx.globalAlpha = 0.3;
|
||||
ctx.drawImage(canvas, -1, 0, canvas.width, canvas.height);
|
||||
ctx.drawImage(canvas, 1, 0, canvas.width, canvas.height);
|
||||
ctx.drawImage(canvas, 0, -1, canvas.width, canvas.height);
|
||||
ctx.drawImage(canvas, 0, 1, canvas.width, canvas.height);
|
||||
ctx.globalAlpha = 1.0;
|
||||
canvas.toBlob(blob => blob ? resolve(blob) : reject(new Error('toBlob failed')), 'image/png');
|
||||
};
|
||||
img.onerror = () => reject(new Error('Image load failed'));
|
||||
img.src = URL.createObjectURL(file);
|
||||
});
|
||||
|
||||
// Then pass to Tesseract
|
||||
const { data: { text } } = await Tesseract.recognize(preprocessed, 'eng', { ... });
|
||||
```
|
||||
|
||||
**Why this works**: Tesseract's default DPI assumption is 70. Doubling the pixel dimensions effectively doubles the perceived DPI. The unsharp mask enhances digit edges that Tesseract's LSTM models rely on for character discrimination.
|
||||
|
||||
**Limitation**: If preprocessing alone doesn't fix time-digit recognition (narrow columns with very small fonts), the LLM cannot compensate — it only sees what Tesseract outputs. In that case, consider cropping the time column and OCR-ing it separately at 4x upscale.
|
||||
|
||||
## Fallback
|
||||
|
||||
Always have a deterministic fallback (regex parser, manual input) for when the LLM server is unreachable. The fetch should be wrapped in try/catch:
|
||||
|
||||
```javascript
|
||||
try {
|
||||
appointments = await llmParse(text);
|
||||
} catch (err) {
|
||||
console.warn('LLM unavailable, falling back:', err.message);
|
||||
appointments = ruleBasedParse(text);
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,474 @@
|
||||
# GGUF Troubleshooting Guide
|
||||
|
||||
## Installation Issues
|
||||
|
||||
### Build Fails
|
||||
|
||||
**Error**: `make: *** No targets specified and no makefile found`
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Ensure you're in llama.cpp directory
|
||||
cd llama.cpp
|
||||
make
|
||||
```
|
||||
|
||||
**Error**: `fatal error: cuda_runtime.h: No such file or directory`
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Install CUDA toolkit
|
||||
# Ubuntu
|
||||
sudo apt install nvidia-cuda-toolkit
|
||||
|
||||
# Or set CUDA path
|
||||
export CUDA_PATH=/usr/local/cuda
|
||||
export PATH=$CUDA_PATH/bin:$PATH
|
||||
make GGML_CUDA=1
|
||||
```
|
||||
|
||||
### Python Bindings Issues
|
||||
|
||||
**Error**: `ERROR: Failed building wheel for llama-cpp-python`
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Install build dependencies
|
||||
pip install cmake scikit-build-core
|
||||
|
||||
# For CUDA support
|
||||
CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir
|
||||
|
||||
# For Metal (macOS)
|
||||
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --force-reinstall --no-cache-dir
|
||||
```
|
||||
|
||||
**Error**: `ImportError: libcudart.so.XX: cannot open shared object file`
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Add CUDA libraries to path
|
||||
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
|
||||
|
||||
# Or reinstall with correct CUDA version
|
||||
pip uninstall llama-cpp-python
|
||||
CUDACXX=/usr/local/cuda/bin/nvcc CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python
|
||||
```
|
||||
|
||||
## Conversion Issues
|
||||
|
||||
### Model Not Supported
|
||||
|
||||
**Error**: `KeyError: 'model.embed_tokens.weight'`
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Check model architecture
|
||||
python -c "from transformers import AutoConfig; print(AutoConfig.from_pretrained('./model').architectures)"
|
||||
|
||||
# Use appropriate conversion script
|
||||
# For most models:
|
||||
python convert_hf_to_gguf.py ./model --outfile model.gguf
|
||||
|
||||
# For older models, check if legacy script needed
|
||||
```
|
||||
|
||||
### Vocabulary Mismatch
|
||||
|
||||
**Error**: `RuntimeError: Vocabulary size mismatch`
|
||||
|
||||
**Fix**:
|
||||
```python
|
||||
# Ensure tokenizer matches model
|
||||
from transformers import AutoTokenizer, AutoModelForCausalLM
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("./model")
|
||||
model = AutoModelForCausalLM.from_pretrained("./model")
|
||||
|
||||
print(f"Tokenizer vocab size: {len(tokenizer)}")
|
||||
print(f"Model vocab size: {model.config.vocab_size}")
|
||||
|
||||
# If mismatch, resize embeddings before conversion
|
||||
model.resize_token_embeddings(len(tokenizer))
|
||||
model.save_pretrained("./model-fixed")
|
||||
```
|
||||
|
||||
### Out of Memory During Conversion
|
||||
|
||||
**Error**: `torch.cuda.OutOfMemoryError` during conversion
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Use CPU for conversion
|
||||
CUDA_VISIBLE_DEVICES="" python convert_hf_to_gguf.py ./model --outfile model.gguf
|
||||
|
||||
# Or use low memory mode
|
||||
python convert_hf_to_gguf.py ./model --outfile model.gguf --outtype f16
|
||||
```
|
||||
|
||||
## Quantization Issues
|
||||
|
||||
### Wrong Output File Size
|
||||
|
||||
**Problem**: Quantized file is larger than expected
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# Verify quantization type
|
||||
./llama-cli -m model.gguf --verbose
|
||||
|
||||
# Expected sizes for 7B model:
|
||||
# Q4_K_M: ~4.1 GB
|
||||
# Q5_K_M: ~4.8 GB
|
||||
# Q8_0: ~7.2 GB
|
||||
# F16: ~13.5 GB
|
||||
```
|
||||
|
||||
### Quantization Crashes
|
||||
|
||||
**Error**: `Segmentation fault` during quantization
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Increase stack size
|
||||
ulimit -s unlimited
|
||||
|
||||
# Or use less threads
|
||||
./llama-quantize -t 4 model-f16.gguf model-q4.gguf Q4_K_M
|
||||
```
|
||||
|
||||
### Poor Quality After Quantization
|
||||
|
||||
**Problem**: Model outputs gibberish after quantization
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Use importance matrix**:
|
||||
```bash
|
||||
# Generate imatrix with good calibration data
|
||||
./llama-imatrix -m model-f16.gguf \
|
||||
-f wiki_sample.txt \
|
||||
--chunk 512 \
|
||||
-o model.imatrix
|
||||
|
||||
# Quantize with imatrix
|
||||
./llama-quantize --imatrix model.imatrix \
|
||||
model-f16.gguf model-q4_k_m.gguf Q4_K_M
|
||||
```
|
||||
|
||||
2. **Try higher precision**:
|
||||
```bash
|
||||
# Use Q5_K_M or Q6_K instead of Q4
|
||||
./llama-quantize model-f16.gguf model-q5_k_m.gguf Q5_K_M
|
||||
```
|
||||
|
||||
3. **Check original model**:
|
||||
```bash
|
||||
# Test FP16 version first
|
||||
./llama-cli -m model-f16.gguf -p "Hello, how are you?" -n 50
|
||||
```
|
||||
|
||||
## Inference Issues
|
||||
|
||||
### Slow Generation
|
||||
|
||||
**Problem**: Generation is slower than expected
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Enable GPU offload**:
|
||||
```bash
|
||||
./llama-cli -m model.gguf -ngl 35 -p "Hello"
|
||||
```
|
||||
|
||||
2. **Optimize batch size**:
|
||||
```python
|
||||
llm = Llama(
|
||||
model_path="model.gguf",
|
||||
n_batch=512, # Increase for faster prompt processing
|
||||
n_gpu_layers=35
|
||||
)
|
||||
```
|
||||
|
||||
3. **Use appropriate threads**:
|
||||
```bash
|
||||
# Match physical cores, not logical
|
||||
./llama-cli -m model.gguf -t 8 -p "Hello"
|
||||
```
|
||||
|
||||
4. **Enable Flash Attention** (if supported):
|
||||
```bash
|
||||
./llama-cli -m model.gguf -ngl 35 --flash-attn -p "Hello"
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
**Error**: `CUDA out of memory` or system freeze
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Reduce GPU layers**:
|
||||
```python
|
||||
# Start low and increase
|
||||
llm = Llama(model_path="model.gguf", n_gpu_layers=10)
|
||||
```
|
||||
|
||||
2. **Use smaller quantization**:
|
||||
```bash
|
||||
./llama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M
|
||||
```
|
||||
|
||||
3. **Reduce context length**:
|
||||
```python
|
||||
llm = Llama(
|
||||
model_path="model.gguf",
|
||||
n_ctx=2048, # Reduce from 4096
|
||||
n_gpu_layers=35
|
||||
)
|
||||
```
|
||||
|
||||
4. **Quantize KV cache**:
|
||||
```python
|
||||
llm = Llama(
|
||||
model_path="model.gguf",
|
||||
type_k=2, # Q4_0 for K cache
|
||||
type_v=2, # Q4_0 for V cache
|
||||
n_gpu_layers=35
|
||||
)
|
||||
```
|
||||
|
||||
### Garbage Output
|
||||
|
||||
**Problem**: Model outputs random characters or nonsense
|
||||
|
||||
**Diagnose**:
|
||||
```python
|
||||
# Check model loading
|
||||
llm = Llama(model_path="model.gguf", verbose=True)
|
||||
|
||||
# Test with simple prompt
|
||||
output = llm("1+1=", max_tokens=5, temperature=0)
|
||||
print(output)
|
||||
```
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Check model integrity**:
|
||||
```bash
|
||||
# Verify GGUF file
|
||||
./llama-cli -m model.gguf --verbose 2>&1 | head -50
|
||||
```
|
||||
|
||||
2. **Use correct chat format**:
|
||||
```python
|
||||
llm = Llama(
|
||||
model_path="model.gguf",
|
||||
chat_format="llama-3" # Match your model: chatml, mistral, etc.
|
||||
)
|
||||
```
|
||||
|
||||
3. **Check temperature**:
|
||||
```python
|
||||
# Use lower temperature for deterministic output
|
||||
output = llm("Hello", max_tokens=50, temperature=0.1)
|
||||
```
|
||||
|
||||
### Token Issues
|
||||
|
||||
**Error**: `RuntimeError: unknown token` or encoding errors
|
||||
|
||||
**Fix**:
|
||||
```python
|
||||
# Ensure UTF-8 encoding
|
||||
prompt = "Hello, world!".encode('utf-8').decode('utf-8')
|
||||
output = llm(prompt, max_tokens=50)
|
||||
```
|
||||
|
||||
## CUDA Build Issues
|
||||
|
||||
### Linker errors for cublas symbols when nvcc and CUDA toolkit versions differ
|
||||
|
||||
**Error**: `undefined reference to cublasGemmEx@libcublas.so.12` (and similar cublas symbols) when building with `-DGGML_CUDA=ON`.
|
||||
|
||||
**Cause**: nvcc version (e.g., 12.4 from `nvidia-cuda-toolkit` apt package) does not match the installed CUDA toolkit libraries (e.g., CUDA 13.1 from `/usr/local/cuda-13.1`). The versioned symbols in libcublas.so.12 from CUDA 13.1 may not match what the CUDA 12.4 headers expect.
|
||||
|
||||
**Diagnose**:
|
||||
```bash
|
||||
nvcc --version | grep release
|
||||
ls /usr/local/cuda*/lib64/libcublas* /usr/lib/x86_64-linux-gnu/libcublas.so.*
|
||||
readlink -f /usr/lib/x86_64-linux-gnu/libcublas.so.12
|
||||
```
|
||||
|
||||
**Fix options**:
|
||||
|
||||
1. **Vulkan backend (recommended sidestep)**: Build with Vulkan instead of CUDA — works on NVIDIA, AMD, and Intel GPUs with no CUDA toolkit needed. See `references/vulkan-gpu-backend.md`.
|
||||
```bash
|
||||
cmake -B build -DGGML_VULKAN=ON
|
||||
cmake --build build -j8
|
||||
# Run with: -ngl 99
|
||||
```
|
||||
|
||||
2. **CPU-only**: Build without GPU acceleration if latency is acceptable for the model size:
|
||||
```bash
|
||||
cmake -B build -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --config Release -j4
|
||||
```
|
||||
|
||||
For small models (1-3B parameters), CPU inference at ~30 tok/s is often sufficient. The GPU speedup is worth pursuing only if latency is critical or the model is larger.
|
||||
|
||||
## Server Issues
|
||||
|
||||
### Connection Refused
|
||||
|
||||
**Error**: `Connection refused` when accessing server
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Bind to all interfaces
|
||||
./llama-server -m model.gguf --host 0.0.0.0 --port 8080
|
||||
|
||||
# Check if port is in use
|
||||
lsof -i :8080
|
||||
```
|
||||
|
||||
### Server Crashes Under Load
|
||||
|
||||
**Problem**: Server crashes with multiple concurrent requests
|
||||
|
||||
**Solutions**:
|
||||
|
||||
1. **Limit parallelism**:
|
||||
```bash
|
||||
./llama-server -m model.gguf \
|
||||
--parallel 2 \
|
||||
-c 4096 \
|
||||
--cont-batching
|
||||
```
|
||||
|
||||
2. **Add request timeout**:
|
||||
```bash
|
||||
./llama-server -m model.gguf --timeout 300
|
||||
```
|
||||
|
||||
3. **Monitor memory**:
|
||||
```bash
|
||||
watch -n 1 nvidia-smi # For GPU
|
||||
watch -n 1 free -h # For RAM
|
||||
```
|
||||
|
||||
### API Compatibility Issues
|
||||
|
||||
**Problem**: OpenAI client not working with server
|
||||
|
||||
**Fix**:
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
# Use correct base URL format
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:8080/v1", # Include /v1
|
||||
api_key="not-needed"
|
||||
)
|
||||
|
||||
# Use correct model name
|
||||
response = client.chat.completions.create(
|
||||
model="local", # Or the actual model name
|
||||
messages=[{"role": "user", "content": "Hello"}]
|
||||
)
|
||||
```
|
||||
|
||||
## Apple Silicon Issues
|
||||
|
||||
### Metal Not Working
|
||||
|
||||
**Problem**: Metal acceleration not enabled
|
||||
|
||||
**Check**:
|
||||
```bash
|
||||
# Verify Metal support
|
||||
./llama-cli -m model.gguf --verbose 2>&1 | grep -i metal
|
||||
```
|
||||
|
||||
**Fix**:
|
||||
```bash
|
||||
# Rebuild with Metal
|
||||
make clean
|
||||
make GGML_METAL=1
|
||||
|
||||
# Python bindings
|
||||
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python --force-reinstall
|
||||
```
|
||||
|
||||
### Incorrect Memory Usage on M1/M2
|
||||
|
||||
**Problem**: Model uses too much unified memory
|
||||
|
||||
**Fix**:
|
||||
```python
|
||||
# Offload all layers for Metal
|
||||
llm = Llama(
|
||||
model_path="model.gguf",
|
||||
n_gpu_layers=99, # Offload everything
|
||||
n_threads=1 # Metal handles parallelism
|
||||
)
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
### Enable Verbose Output
|
||||
|
||||
```bash
|
||||
# CLI verbose mode
|
||||
./llama-cli -m model.gguf --verbose -p "Hello" -n 50
|
||||
|
||||
# Python verbose
|
||||
llm = Llama(model_path="model.gguf", verbose=True)
|
||||
```
|
||||
|
||||
### Check Model Metadata
|
||||
|
||||
```bash
|
||||
# View GGUF metadata
|
||||
./llama-cli -m model.gguf --verbose 2>&1 | head -100
|
||||
```
|
||||
|
||||
### Validate GGUF File
|
||||
|
||||
```python
|
||||
import struct
|
||||
|
||||
def validate_gguf(filepath):
|
||||
with open(filepath, 'rb') as f:
|
||||
magic = f.read(4)
|
||||
if magic != b'GGUF':
|
||||
print(f"Invalid magic: {magic}")
|
||||
return False
|
||||
|
||||
version = struct.unpack('<I', f.read(4))[0]
|
||||
print(f"GGUF version: {version}")
|
||||
|
||||
tensor_count = struct.unpack('<Q', f.read(8))[0]
|
||||
metadata_count = struct.unpack('<Q', f.read(8))[0]
|
||||
print(f"Tensors: {tensor_count}, Metadata: {metadata_count}")
|
||||
|
||||
return True
|
||||
|
||||
validate_gguf("model.gguf")
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
1. **GitHub Issues**: https://github.com/ggml-org/llama.cpp/issues
|
||||
2. **Discussions**: https://github.com/ggml-org/llama.cpp/discussions
|
||||
3. **Reddit**: r/LocalLLaMA
|
||||
|
||||
### Reporting Issues
|
||||
|
||||
Include:
|
||||
- llama.cpp version/commit hash
|
||||
- Build command used
|
||||
- Model name and quantization
|
||||
- Full error message/stack trace
|
||||
- Hardware: CPU/GPU model, RAM, VRAM
|
||||
- OS version
|
||||
- Minimal reproduction steps
|
||||
@@ -0,0 +1,63 @@
|
||||
# VRAM Budget & Model Sizing Guide
|
||||
|
||||
## Formula for Q4_K_M
|
||||
|
||||
VRAM ≈ params_in_billions × 0.58 + context_vram (1–3 GB for 8K–32K context)
|
||||
|
||||
**Tight fit check:** If a model loads but crashes on the first prompt, it's VRAM-starved. Drop one size tier.
|
||||
|
||||
## Model Size Reference
|
||||
|
||||
Approximate VRAM at **Q4_K_M** (Ollama default):
|
||||
|
||||
| Params | Weight size | Total VRAM | Fits in |
|
||||
|--------|-------------|-----------|---------|
|
||||
| 1B–3B | 0.6–1.8 GB | 2–3 GB | Any GPU |
|
||||
| 7B–8B | 4–5 GB | 5–7 GB | 6 GB+, comfortable on 8 GB |
|
||||
| 12B–14B | 7–9 GB | 9–12 GB | 11 GB+ |
|
||||
| 22B–24B | 12–14 GB | 14–17 GB | 16 GB+ |
|
||||
| 32B–35B | 18–21 GB | 20–24 GB | 24 GB |
|
||||
| 70B–72B | 38–42 GB | 42–48 GB | 48 GB+ or dual GPU |
|
||||
|
||||
## Quant Upsizing
|
||||
|
||||
| Quant | Multiplier | 7B model | 14B model | 34B model |
|
||||
|-------|-----------|----------|-----------|-----------|
|
||||
| Q2_K | ×0.33 | 2.3 GB | 4.6 GB | 11.2 GB |
|
||||
| Q3_K_M | ×0.40 | 2.8 GB | 5.6 GB | 13.6 GB |
|
||||
| Q4_K_M | ×0.58 | 4.1 GB | 8.1 GB | 19.7 GB |
|
||||
| Q5_K_M | ×0.68 | 4.8 GB | 9.5 GB | 23.1 GB |
|
||||
| Q6_K | ×0.80 | 5.6 GB | 11.2 GB | 27.2 GB |
|
||||
| Q8_0 | ×1.00 | 7.0 GB | 14.0 GB | 34.0 GB |
|
||||
|
||||
## Known-Good GPU Combos
|
||||
|
||||
| GPU | VRAM | Best LLM (Q4_K_M) | Best Vision |
|
||||
|-----|------|--------------------|-------------|
|
||||
| RTX 3060 | 12 GB | qwen2.5:14b or mistral-nemo:12b | llava:13b or llava-llama3:8b |
|
||||
| RTX 2080 Ti | 11 GB | qwen2.5:14b (tight) or mistral-nemo:12b | llava:13b (tight) or llava-llama3:8b |
|
||||
| RTX 3090 | 24 GB | qwen2.5:32b or llama3:70b (Q3) | llava:34b or llama3.2-vision:11b |
|
||||
| RTX 4090 | 24 GB | Same as 3090 | Same |
|
||||
| RTX 4070 | 12 GB | Same as 3060 | Same |
|
||||
|
||||
## Ollama GPU Investigation
|
||||
|
||||
When the user asks "what's using my GPU":
|
||||
|
||||
1. `nvidia-smi` — all GPU processes with PIDs and VRAM usage
|
||||
2. `ps -p <PID> -o pid,args --no-headers` — which model and port each process runs
|
||||
3. `curl -s http://localhost:11434/api/ps | python3 -m json.tool` — Ollama model details, quant, expiry
|
||||
4. `journalctl -u ollama --since "5 min ago" --no-pager` — recent Ollama activity
|
||||
|
||||
The `expires_at` field tells when Ollama auto-unloads (default 5 min idle).
|
||||
|
||||
## Removing Stale llama.cpp Server
|
||||
|
||||
```bash
|
||||
kill <PID>
|
||||
rm -rf /path/to/build /path/to/model.gguf
|
||||
sudo systemctl stop llama-server 2>/dev/null
|
||||
sudo systemctl disable llama-server 2>/dev/null
|
||||
sudo rm /etc/systemd/system/llama-server.service
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
@@ -0,0 +1,140 @@
|
||||
# Vulkan GPU Backend for llama.cpp
|
||||
|
||||
Build and deploy llama.cpp with GPU acceleration via Vulkan — no CUDA toolkit required. Works on NVIDIA, AMD, and Intel GPUs with Vulkan drivers.
|
||||
|
||||
## When to use Vulkan
|
||||
|
||||
- **CUDA toolkit version mismatch** (nvcc 12.4 vs CUDA 13.1 libs) — Vulkan sidesteps it entirely
|
||||
- No CUDA toolkit installed and you don't want to install one
|
||||
- AMD or Intel GPU (ROCm not set up)
|
||||
- Want a single backend that works across GPU vendors
|
||||
|
||||
## Performance
|
||||
|
||||
Vulkan delivers 80-90% of CUDA inference speed for llama.cpp. For a 7B Q2_K model on a GTX 1050 Ti (4GB), expect ~11-12s per request at 4096 context.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install -y libvulkan-dev glslc glslang-dev glslang-tools libglm-dev
|
||||
|
||||
# glslc is the GLSL→SPIR-V shader compiler (critical — cmake fails with "Could NOT find Vulkan (missing: glslc)" without it)
|
||||
# It's available as the standalone 'glslc' package on Ubuntu 24.04+, or bundled in 'libshaderc-dev'
|
||||
# If 'glslc' package not found, use: sudo apt-get install -y libshaderc-dev
|
||||
# libglm-dev provides GLM math headers needed by the Vulkan shader compilation step
|
||||
|
||||
# Check which package provides glslc on your distro:
|
||||
# apt-cache search glslc # should show both 'glslc' and 'libshaderc-dev'
|
||||
# dpkg -S $(which glslc) # find installed package
|
||||
```
|
||||
|
||||
Also note: switching backends (CPU-only ↔ Vulkan) requires a fresh cmake configure. The cached build uses the previous backend:
|
||||
|
||||
```bash
|
||||
# If you built CPU-only first, then want Vulkan:
|
||||
cd llama.cpp
|
||||
cmake -B build -DGGML_VULKAN=ON # reconfigures from scratch
|
||||
cmake --build build -j8
|
||||
|
||||
# If the cmake cache had GGML_VULKAN=OFF from a previous build,
|
||||
# you must explicitly set it ON — cmake remembers the old value
|
||||
```
|
||||
|
||||
Verify Vulkan driver is loaded:
|
||||
```bash
|
||||
lsmod | grep nvidia # or amdgpu for AMD
|
||||
```
|
||||
|
||||
## Build from source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggml-org/llama.cpp
|
||||
cd llama.cpp
|
||||
cmake -B build -DGGML_VULKAN=ON
|
||||
cmake --build build -j8
|
||||
```
|
||||
|
||||
Verify Vulkan was detected:
|
||||
```bash
|
||||
./build/bin/llama-server --help 2>&1 | grep -i vulkan
|
||||
# Should show: "ggml_vulkan: Found 1 Vulkan devices:"
|
||||
# Should list your GPU model
|
||||
```
|
||||
|
||||
## Launch with full GPU offload
|
||||
|
||||
```bash
|
||||
./build/bin/llama-server \
|
||||
-m /path/to/model.gguf \
|
||||
-c 4096 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8081 \
|
||||
-ngl 99 \
|
||||
-t 8
|
||||
```
|
||||
|
||||
- `-ngl 99`: offload all layers to GPU (use 99 to mean "everything")
|
||||
- `-t 8`: CPU threads for any remaining CPU work (KV cache management, tokenization)
|
||||
- If model doesn't fully fit in VRAM, reduce `-ngl` (e.g., `-ngl 22` for ~80% layers on GPU)
|
||||
|
||||
## Systemd service
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=llama.cpp server with Vulkan GPU
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=ray
|
||||
WorkingDirectory=/home/ray
|
||||
ExecStart=/home/ray/llama.cpp-build/build/bin/llama-server \
|
||||
-m /home/ray/models/Qwen2.5-7B-Instruct-Q2_K.gguf \
|
||||
-c 4096 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8081 \
|
||||
-ngl 99 \
|
||||
-t 8
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
## VRAM sizing
|
||||
|
||||
How to estimate if a model fits in VRAM:
|
||||
|
||||
| Model | Quant | File size | VRAM (~) | Fits 4GB? |
|
||||
|---|---|---|---|---|
|
||||
| Qwen2.5-7B | Q2_K | 3.0 GB | 3.2 GB | ✅ |
|
||||
| Qwen2.5-7B | Q3_K_M | 3.5 GB | 3.7 GB | ✅ |
|
||||
| Qwen2.5-7B | Q4_K_M | 4.7 GB | 5.0 GB | ❌ (partial offload) |
|
||||
| Qwen2.5-3B | Q4_K_M | 1.9 GB | 2.1 GB | ✅ |
|
||||
| Qwen2.5-3B | Q8_0 | 3.3 GB | 3.5 GB | ✅ |
|
||||
|
||||
Rule of thumb: VRAM ≈ file size + 200-300MB for KV cache at 4096 context.
|
||||
|
||||
## Detection and verification
|
||||
|
||||
```bash
|
||||
# List Vulkan devices detected by llama.cpp
|
||||
./build/bin/llama-server --help 2>&1 | grep "ggml_vulkan"
|
||||
|
||||
# Example output:
|
||||
# ggml_vulkan: Found 1 Vulkan devices:
|
||||
# ggml_vulkan: 0 = NVIDIA GeForce GTX 1050 Ti (NVIDIA) | uma: 0 | fp16: 0 | warp size: 32
|
||||
```
|
||||
|
||||
Check GPU memory usage during inference:
|
||||
```bash
|
||||
nvidia-smi # NVIDIA
|
||||
# or
|
||||
radeontop # AMD
|
||||
```
|
||||
|
||||
## Switching back to CPU-only
|
||||
|
||||
Remove `-ngl 99` from the command. The same Vulkan-built binary works for CPU — it just won't offload any layers.
|
||||
Reference in New Issue
Block a user