ray
106 lines
4.9 KiB
Markdown
106 lines
4.9 KiB
Markdown
---
|
|
name: hermes-local-providers
|
|
description: "Configure Hermes Agent to use self-hosted, local LLM providers (Ollama, vLLM, llama.cpp, etc.) via OpenAI-compatible custom endpoints."
|
|
version: 1.0.0
|
|
author: agent
|
|
created_by: agent
|
|
metadata:
|
|
hermes:
|
|
tags: [hermes, ollama, local-models, configuration, self-hosted, vllm, llm-serving]
|
|
related_skills: [hermes-agent, llama-cpp, serving-llms-vllm, docker-gpu-acceleration]
|
|
---
|
|
|
|
# Hermes Local Providers
|
|
|
|
Configure Hermes Agent to use self-hosted / local LLMs that expose an OpenAI-compatible API endpoint.
|
|
|
|
## Supported Backends
|
|
|
|
| Backend | Typical Base URL | Auth | Notes |
|
|
|---------|-----------------|------|-------|
|
|
| Ollama | `http://localhost:11434/v1` | Placeholder | Most common for local use |
|
|
| vLLM | `http://localhost:8000/v1` | Optional API key | Production-scale serving |
|
|
| llama.cpp | `http://localhost:8080/v1` | None | Lightweight GGUF inference |
|
|
|
|
## Configuration
|
|
|
|
### 1. Set the custom provider
|
|
|
|
```bash
|
|
hermes config set model.provider custom
|
|
hermes config set model.base_url http://localhost:11434/v1
|
|
hermes config set model.api_key ollama
|
|
hermes config set model.default qwen2.5:14b
|
|
```
|
|
|
|
The provider name is **`custom`** — no suffix, no prefix. Hermes recognizes `custom` as a reserved provider that reads `model.base_url` and `model.api_key` directly from the config. Values like `custom:ollama` or `custom:vllm` will be rejected as "Unknown provider."
|
|
|
|
Ollama doesn't need a real API key, but `model.api_key` must be set to something non-empty. `ollama` (or any dummy string) works.
|
|
|
|
### 2. ⚠️ Verify the config persisted
|
|
|
|
`hermes config set` on `model.*` keys may report success (`✓ Set model.key = value`) but NOT actually write the value to `~/.hermes/config.yaml`. This is a known pitfall — always verify:
|
|
|
|
```bash
|
|
grep -A6 '^model:' ~/.hermes/config.yaml
|
|
```
|
|
|
|
Expected result:
|
|
```yaml
|
|
model:
|
|
default: qwen2.5:14b
|
|
provider: custom
|
|
base_url: http://localhost:11434/v1
|
|
api_key: ollama
|
|
```
|
|
|
|
> ⚠️ The provider must be **`custom`** (bare) — NOT `custom:ollama`, `custom:vllm`, or any suffix. Hermes only recognizes the bare `custom` as a reserved provider for user-defined endpoints. `custom:ollama` will be rejected as "Unknown provider."
|
|
|
|
If the old values are still showing, re-run the `hermes config set` commands and re-check. Retrying usually works; the silent failure appears to be intermittent.
|
|
|
|
### 3. Test the API independently
|
|
|
|
Before restarting the gateway, confirm the local backend is reachable:
|
|
|
|
```bash
|
|
curl -s http://localhost:11434/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model":"qwen2.5:14b","messages":[{"role":"user","content":"Say hello in 3 words"}],"stream":false}'
|
|
```
|
|
|
|
If this fails, Ollama/vLLM/llama.cpp isn't running or the model isn't loaded.
|
|
|
|
### 4. Restart the gateway
|
|
|
|
```bash
|
|
hermes gateway restart
|
|
```
|
|
|
|
Config changes only take effect after a gateway restart. For CLI sessions, exit and relaunch.
|
|
|
|
## Pitfalls
|
|
|
|
- **Config set silent failure**: `hermes config set` for `model.*` keys (`provider`, `base_url`, `api_key`, `default`) may report "✓ Set" but not persist. Always grep the config file to confirm before assuming the change took effect.
|
|
- **Auth placeholder required**: A local Ollama endpoint doesn't authenticate, but Hermes requires `model.api_key` to be non-empty. Any dummy value works.
|
|
- **Gateway restart required**: Unlike some config changes, switching the model provider requires a full gateway restart. `/reset` or `/model` in-session won't pick up the new provider settings.
|
|
- **Auxiliary models stay unchanged**: Vision, compression, web extraction, title generation, and all other `auxiliary.*` subsystems keep their own provider config. Switching the main model to Ollama doesn't affect these — they'll continue using whatever provider/API key they were configured with (DeepSeek, OpenRouter, etc.).
|
|
- **Delegation has its own config**: The `delegation.*` section (`delegation.model`, `delegation.provider`, `delegation.base_url`) controls subagent models. To route subagents through the local provider too, set those separately:
|
|
```bash
|
|
hermes config set delegation.provider custom
|
|
hermes config set delegation.base_url http://localhost:11434/v1
|
|
hermes config set delegation.api_key ollama
|
|
hermes config set delegation.model qwen2.5:14b
|
|
```
|
|
- **Model must be downloaded**: `ollama list` shows installed models. If you set `model.default` to a model that isn't pulled, Ollama will pull it on first request (slow first call) or error if auto-pull is disabled.
|
|
|
|
## Verification
|
|
|
|
After restart, send a message to the agent. If you get "provider authentication error":
|
|
|
|
1. Check `~/.hermes/config.yaml` — the old provider values may have persisted
|
|
2. Re-run the `hermes config set` commands
|
|
3. Verify with `grep -A5 '^model:' ~/.hermes/config.yaml`
|
|
4. Restart gateway again
|
|
|
|
The "authentication error" is misleading — it usually means the API key/base URL weren't received by Hermes, which happens when `hermes config set` failed to persist.
|