initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,133 @@
+---
+name: hermes-dashboard
+description: "Deploy and extend the Hermes Dashboard — a browser control panel for managing config, models, tools, skills, cron jobs, and logs without the CLI."
+version: 1.0.0
+author: Hermes Agent
+license: MIT
+platforms: [linux]
+metadata:
+  hermes:
+    tags: [hermes, dashboard, monitoring, control-panel, docker]
+    related_skills: [hermes-webui, hermes-agent, server-health-check]
+---
+
+# Hermes Dashboard
+
+A Flask-based control panel that complements the Hermes Web UI (chat interface). While the Web UI provides chat + session browsing, the Dashboard provides **management** — config editing, model switching, tool toggling, skill browsing, cron job control, and log viewing.
+
+## Architecture Decision: No CLI Dependency
+
+The dashboard reads Hermes state **directly from the filesystem** rather than shelling out to the `hermes` CLI. This avoids the venv shebang problem (host Python paths break in Docker containers) and keeps the image lightweight.
+
+| Data | Source |
+|------|--------|
+| Config | `config.yaml` (read/write via PyYAML) |
+| Cron jobs | `cron/jobs.json` (read/write) |
+| Skills | Filesystem scan of `skills/` for `SKILL.md` |
+| Logs | `logs/gateway.log`, `logs/error.log` (tail) |
+| System status | `systemctl`, `curl` health checks |
+
+## Deployment
+
+### Directory Layout
+```
+~/docker/hermes-dashboard/
+├── app.py              # Flask backend (all routes)
+├── templates/          # Jinja2 HTML (Bootstrap 5 dark theme)
+│   ├── base.html       # Nav + layout shell
+│   ├── login.html      # Password auth
+│   ├── index.html      # Overview with live status
+│   ├── config.html     # YAML editor + quick setter
+│   ├── models.html     # Model/provider switcher
+│   ├── tools.html      # Toolset enable/disable
+│   ├── skills.html     # Skill browser + installer
+│   ├── cron.html       # Cron job list + toggle/trigger
+│   └── logs.html       # Log viewer with auto-refresh
+├── Dockerfile          # python:3.11-slim + Flask + PyYAML
+├── docker-compose.yml  # Volume mounts ~/.hermes, port 5000, env_file
+├── requirements.txt    # flask, pyyaml, werkzeug
+└── .env                # DASHBOARD_PASSWORD, DASHBOARD_SECRET_KEY
+```
+
+### Quick Deploy
+```bash
+# 1. Create the directory and files (see references/dashboard-app.py for full source)
+mkdir -p ~/docker/hermes-dashboard/{templates,static}
+
+# 2. Set password (generate hash with werkzeug)
+python3 -c "from werkzeug.security import generate_password_hash; print(generate_password_hash('your-password'))"
+
+# 3. Launch
+cd ~/docker/hermes-dashboard
+docker compose up -d --build
+
+# 4. Add nginx reverse proxy (reuse existing certbot SSL cert)
+#    See existing nginx configs under /etc/nginx/sites-enabled/ for pattern
+```
+
+### Nginx Config Pattern
+```nginx
+server {
+    listen 3445 ssl;  # Pick next available port
+    server_name your-domain.duckdns.org;
+    ssl_certificate /etc/letsencrypt/live/your-domain/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/your-domain/privkey.pem;
+    location / {
+        proxy_pass http://127.0.0.1:8788;
+        proxy_set_header Host $host;
+    }
+}
+```
+
+## Cron Job Management
+
+Cron jobs are read from `~/.hermes/cron/jobs.json`. The JSON structure uses `id` (not `job_id`) as the primary key, with `schedule_display` for human-readable schedules and `enabled` as a boolean.
+
+Toggle pause/resume by flipping `enabled` in jobs.json and writing it back. The scheduler picks up changes on next tick.
+
+For "Run Now", write a trigger flag at `~/.hermes/cron/.trigger_<id>` with a timestamp. The scheduler polls for these.
+
+## Design System
+
+The UI uses a Linear/Vercel-inspired dark theme with these characteristics:
+
+- **Sidebar navigation** (240px fixed, icons + labels, active state highlight)
+- **Glass-morphism cards** with `#18181b` surface, `#27272a` borders, 12px radius
+- **Stat cards** with gradient top border accent (`#6366f1 → #a855f7`) and pulse-animated status dots
+- **Typography**: Inter for UI, JetBrains Mono for code/logs
+- **No Bootstrap JS/CSS dependency** — pure custom CSS with Bootstrap Icons
+- **Color tokens**: `--accent: #6366f1` (indigo), `--green: #22c55e`, `--red: #ef4444`, `--amber: #f59e0b`
+
+## Pitfalls
+
+- **Password hash with `$` signs in Docker env_file**: Docker Compose interprets `$` as variable substitution. Use `env_file` (not inline `environment:`) and ensure the hash value is quoted or escaped. The dashboard reads the raw password from `DASHBOARD_PASSWORD` and hashes it at startup. If login silently fails (returns 200 with login page, no redirect), the password hash is wrong. Verify with:
+  ```bash
+  docker exec hermes-dashboard python3 -c "
+  from werkzeug.security import check_password_hash, generate_password_hash
+  import os
+  pw = os.environ['DASHBOARD_PASSWORD']
+  print('Password:', pw)
+  print('Verify with fresh hash:', check_password_hash(generate_password_hash(pw), pw))
+  "
+  ```
+  Then test login directly:
+  ```bash
+  curl -sk -c /tmp/cookies.txt -X POST https://localhost:3445/login \
+    -d "password=YOUR_PASSWORD" -w "%{http_code}"
+  # Should return 302 (redirect to /), not 200 (login page)
+  ```
+
+- **Session cookies break on restart**: Flask `secret_key` must be stable. Set `DASHBOARD_SECRET_KEY` in `.env` (not auto-generated) so sessions survive container restarts. Without it, `os.urandom(24).hex()` generates a new secret each restart, invalidating all cookies.
+
+- **`hermes` CLI doesn't work in Docker**: The venv shebang points to host Python paths. Don't try to run `hermes` CLI inside the container — read files directly instead.
+
+- **Toolsets may be list or dict in config.yaml**: The `toolsets` key can be either format. When it's a list (e.g. `['hermes-cli']`), it's a per-platform override — NOT an exhaustive enabled-set. **Treat all known tools as enabled by default when toolsets is a list.** Only explicit dict entries with `enabled: false` should show as disabled. The view function must handle both formats, and the toggle endpoint must convert list→dict before writing. Always define `all_known` BEFORE the conversion block (Python NameError otherwise).
+
+- **Docker Compose `env_file` vs inline `environment:`**: When using `env_file: .env`, Docker Compose reads the file directly without `$` substitution. Inline `environment:` treats `$` as variable references. For bcrypt hashes (which contain `$2b$...`), always use `env_file`.
+
+- **Container recreate needed for image changes**: `docker restart` keeps the old image. Use `docker compose up -d --build --force-recreate` to pick up code changes.
+
+## References
+
+- `references/firecrawl-reddit-limitations.md` — Firecrawl blocks Reddit; RSS workaround for WSB
+- `references/dashboard-app.py` — Full app.py source (or use skill_view to read it)
@@ -0,0 +1,291 @@
+#!/usr/bin/env python3
+"""Hermes Agent Dashboard — Full Control Panel (no CLI dependency)
+
+See SKILL.md for architecture and deployment guide.
+Source: ~/docker/hermes-dashboard/app.py
+"""
+
+import os, json, subprocess, re, yaml, glob, sqlite3
+from datetime import datetime, timezone
+from functools import wraps
+from flask import Flask, render_template, request, jsonify, redirect, url_for, session
+from werkzeug.security import generate_password_hash, check_password_hash
+
+app = Flask(__name__)
+app.secret_key = os.environ.get("DASHBOARD_SECRET_KEY", os.urandom(24).hex())
+HERMES_HOME = os.environ.get("HERMES_HOME", os.path.expanduser("~/.hermes"))
+CONFIG_PATH = os.path.join(HERMES_HOME, "config.yaml")
+LOGS_DIR = os.path.join(HERMES_HOME, "logs")
+SKILLS_DIR = os.path.join(HERMES_HOME, "skills")
+
+PASSWORD_HASH = os.environ.get("DASHBOARD_PASSWORD_HASH")
+if not PASSWORD_HASH:
+    pw = os.environ.get("DASHBOARD_PASSWORD", "hermes")
+    PASSWORD_HASH = generate_password_hash(pw)
+
+# ---- Auth ----
+def login_required(f):
+    @wraps(f)
+    def decorated(*args, **kwargs):
+        if not session.get("authenticated"):
+            return redirect(url_for("login"))
+        return f(*args, **kwargs)
+    return decorated
+
+@app.route("/login", methods=["GET", "POST"])
+def login():
+    if request.method == "POST":
+        if check_password_hash(PASSWORD_HASH, request.form.get("password", "")):
+            session["authenticated"] = True
+            return redirect(url_for("index"))
+        return render_template("login.html", error="Invalid password")
+    return render_template("login.html")
+
+@app.route("/logout")
+def logout():
+    session.clear()
+    return redirect(url_for("login"))
+
+# ---- Helpers ----
+def send_command(cmd, timeout=10):
+    try:
+        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, shell=True)
+        return result.stdout + result.stderr
+    except:
+        return "Command failed"
+
+def get_status():
+    status = {"gateway": "unknown", "api_server": "unknown", "cron_jobs": 0, "disk_usage": "unknown"}
+    gw = send_command("systemctl --user is-active hermes-gateway 2>/dev/null || echo inactive", timeout=3)
+    status["gateway"] = "running" if "active" in gw else "stopped"
+    api = send_command("curl -s -o /dev/null -w '%{http_code}' http://localhost:8642/health 2>/dev/null || echo 000", timeout=3)
+    status["api_server"] = "running" if api.strip() == "200" else "stopped"
+    # Count cron jobs from jobs.json
+    jobs_path = os.path.join(HERMES_HOME, "cron", "jobs.json")
+    if os.path.exists(jobs_path):
+        try:
+            with open(jobs_path) as f:
+                data = json.load(f)
+            status["cron_jobs"] = len(data.get("jobs", []))
+        except:
+            status["cron_jobs"] = "?"
+    du = send_command(f"df -h {HERMES_HOME} | tail -1 | awk '{{print $5}}'", timeout=3)
+    status["disk_usage"] = du.strip()
+    return status
+
+def read_config():
+    try:
+        with open(CONFIG_PATH) as f:
+            return yaml.safe_load(f)
+    except:
+        return {}
+
+def write_config(data):
+    try:
+        with open(CONFIG_PATH, 'w') as f:
+            yaml.dump(data, f, default_flow_style=False, sort_keys=False)
+        return True
+    except:
+        return False
+
+def get_cron_jobs():
+    """Read cron jobs from jobs.json"""
+    jobs_path = os.path.join(HERMES_HOME, "cron", "jobs.json")
+    if not os.path.exists(jobs_path):
+        return []
+    try:
+        with open(jobs_path) as f:
+            data = json.load(f)
+        return data.get("jobs", [])
+    except:
+        return []
+
+def get_skills_list():
+    skills = []
+    if os.path.exists(SKILLS_DIR):
+        for root, dirs, files in os.walk(SKILLS_DIR):
+            if "SKILL.md" in files:
+                rel = os.path.relpath(root, SKILLS_DIR)
+                skill_path = os.path.join(root, "SKILL.md")
+                try:
+                    with open(skill_path) as f:
+                        content = f.read()
+                        name = desc = ""
+                        for line in content.split('\n'):
+                            if line.startswith("name:"): name = line.split(":",1)[1].strip().strip('"')
+                            elif line.startswith("description:"): desc = line.split(":",1)[1].strip().strip('"')
+                        if not name: name = rel
+                except:
+                    name, desc = rel, ""
+                skills.append({"name": name, "path": rel, "description": desc})
+    return sorted(skills, key=lambda s: s["name"].lower())
+
+# ---- Routes ----
+@app.route("/")
+@login_required
+def index():
+    return render_template("index.html", status=get_status())
+
+@app.route("/config")
+@login_required
+def config_view():
+    cfg = read_config()
+    def mask(d, path=""):
+        if isinstance(d, dict): return {k: mask(v, f"{path}.{k}") for k,v in d.items()}
+        if isinstance(d, list): return [mask(v, path) for v in d]
+        if isinstance(d, str) and any(x in path.lower() for x in ["api_key","token","secret","password"]):
+            if len(d) > 8: return d[:4]+"..."+d[-4:]
+        return d
+    return render_template("config.html", config=mask(cfg), raw=yaml.dump(cfg, default_flow_style=False, sort_keys=False))
+
+@app.route("/config/edit", methods=["POST"])
+@login_required
+def config_edit():
+    try:
+        data = yaml.safe_load(request.form.get("yaml_content", ""))
+        ok = write_config(data)
+        return jsonify({"success": ok, "message": "Config updated. Restart gateway." if ok else "Failed"})
+    except yaml.YAMLError as e:
+        return jsonify({"success": False, "message": f"YAML error: {e}"})
+
+@app.route("/config/quick", methods=["POST"])
+@login_required
+def config_quick():
+    key = request.form.get("key")
+    value = request.form.get("value")
+    if not key: return jsonify({"success": False, "message": "Key required"})
+    cfg = read_config()
+    keys = key.split(".")
+    target = cfg
+    for k in keys[:-1]:
+        if k not in target: target[k] = {}
+        target = target[k]
+    try: val = int(value)
+    except ValueError:
+        val = value.lower() == "true" if value.lower() in ("true","false") else value
+    target[keys[-1]] = val
+    return jsonify({"success": write_config(cfg), "message": f"Set {key} = {value}"})
+
+@app.route("/models")
+@login_required
+def models_view():
+    cfg = read_config()
+    return render_template("models.html",
+        current_model=cfg.get("model",{}).get("default","unknown"),
+        current_provider=cfg.get("model",{}).get("provider","unknown"),
+        delegation_model=cfg.get("delegation",{}).get("model",""))
+
+@app.route("/models/switch", methods=["POST"])
+@login_required
+def models_switch():
+    cfg = read_config()
+    provider = request.form.get("provider")
+    model = request.form.get("model")
+    if provider: cfg.setdefault("model",{})["provider"] = provider
+    if model: cfg.setdefault("model",{})["default"] = model
+    return jsonify({"success": write_config(cfg), "message": "Model updated. Restart gateway."})
+
+@app.route("/tools")
+@login_required
+def tools_view():
+    cfg = read_config()
+    toolsets = cfg.get("toolsets", {})
+    all_known = ["web","terminal","file","search","browser","vision","image_gen","video","tts",
+                 "skills","memory","session_search","delegation","cronjob","clarify","messaging",
+                 "todo","spotify","homeassistant","discord","code_execution","kanban"]
+    if isinstance(toolsets, list):
+        enabled_names = set(toolsets)
+        toolsets = {name: {"enabled": name in enabled_names} for name in all_known}
+    enabled = []
+    for t in all_known:
+        ts_cfg = toolsets.get(t, {})
+        is_enabled = ts_cfg.get("enabled", True) if isinstance(ts_cfg, dict) else True
+        enabled.append({"name": t, "enabled": is_enabled})
+    return render_template("tools.html", tools=enabled)
+
+@app.route("/tools/toggle", methods=["POST"])
+@login_required
+def tools_toggle():
+    name = request.form.get("name")
+    action = request.form.get("action")
+    cfg = read_config()
+    cfg.setdefault("toolsets", {}).setdefault(name, {})
+    cfg["toolsets"][name]["enabled"] = (action == "enable")
+    return jsonify({"success": write_config(cfg), "message": f"Toolset '{name}' {action}d"})
+
+@app.route("/skills")
+@login_required
+def skills_view():
+    return render_template("skills.html", skills=get_skills_list())
+
+@app.route("/cron")
+@login_required
+def cron_view():
+    return render_template("cron.html", jobs=get_cron_jobs())
+
+@app.route("/cron/toggle", methods=["POST"])
+@login_required
+def cron_toggle():
+    job_id = request.form.get("job_id")
+    enabled = request.form.get("enabled") == "true"
+    jobs_path = os.path.join(HERMES_HOME, "cron", "jobs.json")
+    try:
+        with open(jobs_path) as f: data = json.load(f)
+        for job in data.get("jobs", []):
+            if job["id"] == job_id:
+                job["enabled"] = enabled
+                break
+        with open(jobs_path, 'w') as f: json.dump(data, f, indent=2)
+        return jsonify({"success": True, "message": f"Job {'enabled' if enabled else 'paused'}"})
+    except Exception as e:
+        return jsonify({"success": False, "message": str(e)})
+
+@app.route("/cron/run", methods=["POST"])
+@login_required
+def cron_run():
+    job_id = request.form.get("job_id")
+    flag = os.path.join(HERMES_HOME, "cron", f".trigger_{job_id}")
+    try:
+        os.makedirs(os.path.dirname(flag), exist_ok=True)
+        with open(flag, 'w') as f: f.write(datetime.now(timezone.utc).isoformat())
+        return jsonify({"success": True, "message": "Job triggered. Should run within 30s."})
+    except Exception as e:
+        return jsonify({"success": False, "message": str(e)})
+
+@app.route("/logs")
+@login_required
+def logs_view():
+    log_type = request.args.get("type", "gateway")
+    lines = request.args.get("lines", "100")
+    log_files = {"gateway": os.path.join(LOGS_DIR,"gateway.log"), "error": os.path.join(LOGS_DIR,"error.log")}
+    content = ""
+    log_path = log_files.get(log_type)
+    if log_path and os.path.exists(log_path):
+        content = send_command(f"tail -n {lines} {log_path}", timeout=5)
+    else:
+        available = [k for k,v in log_files.items() if os.path.exists(v)]
+        content = f"Available: {', '.join(available)}" if available else "No log files found."
+    return render_template("logs.html", content=content, log_type=log_type, lines=lines)
+
+@app.route("/api/status")
+@login_required
+def api_status():
+    return jsonify(get_status())
+
+@app.route("/api/logs/tail")
+@login_required
+def api_logs_tail():
+    log_type = request.args.get("type", "gateway")
+    log_files = {"gateway": os.path.join(LOGS_DIR,"gateway.log"), "error": os.path.join(LOGS_DIR,"error.log")}
+    log_path = log_files.get(log_type)
+    if log_path and os.path.exists(log_path):
+        return jsonify({"content": send_command(f"tail -n 50 {log_path}", timeout=3)})
+    return jsonify({"content": "Log file not found"})
+
+@app.route("/health")
+def health():
+    return jsonify({"status": "ok", "timestamp": datetime.now(timezone.utc).isoformat()})
+
+if __name__ == "__main__":
+    port = int(os.environ.get("DASHBOARD_PORT", 5000))
+    app.run(host="0.0.0.0", port=port, debug=False)
@@ -0,0 +1,68 @@
+# Firecrawl + Reddit Limitations
+
+## Discovery (June 2026)
+
+Firecrawl (firecrawl.dev, `firecrawl-py` SDK) **explicitly blocks Reddit domains**:
+
+```
+HTTP 403: "We apologize for the inconvenience but we do not support this site."
+```
+
+This applies to all Reddit URLs: `old.reddit.com`, `www.reddit.com`, `reddit.com`, and all subreddits.
+
+## Workaround: Reddit RSS Feeds
+
+Reddit provides Atom/RSS feeds at `https://www.reddit.com/r/<subreddit>/.rss`. These are less aggressively blocked than the JSON API.
+
+**Availability (tested June 2026):**
+
+| Subreddit | `.rss` | `.json` API |
+|-----------|--------|-------------|
+| r/wallstreetbets | ✅ 200 | ❌ Blocked |
+| r/stocks | ❌ 403 | ❌ Blocked |
+| r/StockMarket | ❌ 403 | ❌ Blocked |
+| r/pennystocks | ❌ 403 | ❌ Blocked |
+
+Only r/wallstreetbets RSS is accessible. Other financial subreddits block all programmatic access.
+
+## Fetching WSB RSS
+
+```bash
+curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
+  "https://www.reddit.com/r/wallstreetbets/.rss"
+```
+
+Returns Atom XML with post titles, links, and timestamps. Parse for `$TICKER` mentions and sentiment.
+
+## What Doesn't Work
+
+- Firecrawl SDK on any Reddit URL
+- Reddit `.json` API (returns HTML blocks: "whoa there, pardner")
+- `old.reddit.com/r/*/.json` (same block)
+- `api.reddit.com/r/*/hot` (same block)
+- Most subreddit RSS feeds (403 on non-WSB subs)
+
+## Firecrawl Works For
+
+Financial news sites are fine:
+- `finance.yahoo.com` ✅
+- `marketwatch.com` ✅
+- `finviz.com` ✅
+- `news.google.com` ✅
+
+## Python SDK Usage
+
+```python
+from firecrawl import FirecrawlApp
+import os
+
+app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
+
+# Correct: formats is a keyword argument (list), NOT params=dict
+result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
+md = result.get('markdown', '')
+```
+
+Install: `pip install firecrawl-py` (v4.28.2 tested). Free tier: 500 credits/month, 1 credit per scrape.
+
+API key format: `fc-1a44...` (35 chars). Store in `~/.hermes/.env` as `FIRECRAWL_API_KEY`. When reading from `.env`, skip commented lines (grep for lines NOT starting with `#`).
@@ -0,0 +1,20 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    curl ca-certificates \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY . .
+
+ENV DASHBOARD_PORT=5000
+ENV HERMES_HOME=/home/hermes/.hermes
+ENV PYTHONPATH=/home/hermes/.hermes/hermes-agent
+
+EXPOSE 5000
+
+CMD ["python", "app.py"]
@@ -0,0 +1,20 @@
+services:
+  hermes-dashboard:
+    build: .
+    container_name: hermes-dashboard
+    restart: unless-stopped
+    ports:
+      - "127.0.0.1:8788:5000"
+    volumes:
+      - /home/ray/.hermes:/home/hermes/.hermes
+    env_file:
+      - .env
+    environment:
+      - HERMES_HOME=/home/hermes/.hermes
+    networks:
+      - hermes-net
+
+networks:
+  hermes-net:
+    external: true
+    name: hermes-net
@@ -0,0 +1,18 @@
+server {
+    listen 3445 ssl;
+    server_name grajmedia.duckdns.org;
+
+    ssl_certificate /etc/letsencrypt/live/grajmedia.duckdns.org/fullchain.pem;
+    ssl_certificate_key /etc/letsencrypt/live/grajmedia.duckdns.org/privkey.pem;
+    ssl_protocols TLSv1.2 TLSv1.3;
+    ssl_ciphers HIGH:!aNULL:!MD5;
+
+    location / {
+        proxy_pass http://127.0.0.1:8788;
+        proxy_http_version 1.1;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}