initial commit
This commit is contained in:
@@ -0,0 +1,139 @@
|
||||
---
|
||||
name: recurring-briefings
|
||||
description: Set up cron jobs for periodic data-gathering and formatted briefings — weather checks, market updates, monitoring digests, or any recurring "fetch → analyze → deliver" workflow.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
tags: [cron, scheduling, briefings, automation, weather, stocks, monitoring]
|
||||
---
|
||||
|
||||
# Recurring Briefings
|
||||
|
||||
Set up automated daily/weekly briefings that fetch external data, analyze it against user-defined constraints, and deliver formatted reports on a schedule.
|
||||
|
||||
## Core Pattern
|
||||
|
||||
Every recurring briefing follows the same shape:
|
||||
|
||||
```
|
||||
cron job (scheduled) → fetch data (terminal/curl) → analyze → format → deliver (Telegram)
|
||||
```
|
||||
|
||||
## Step 1 — Define the Briefing
|
||||
|
||||
Clarify with the user:
|
||||
|
||||
| Question | Example |
|
||||
|----------|---------|
|
||||
| **What data?** | Weather, stock chatter, news headlines |
|
||||
| **What constraints/filters?** | "After 6pm on weekdays," "most talked-about stocks" |
|
||||
| **When?** | "Every Monday 8 AM," "Weekdays 8:30 AM" |
|
||||
| **Format?** | Bullet list, table-free (Telegram), emoji headers |
|
||||
|
||||
## Step 2 — Create the Cron Job
|
||||
|
||||
Use `cronjob(action='create')` with these settings:
|
||||
|
||||
```yaml
|
||||
schedule: cron expression or human string like "0 8 * * 1" (Mon 8 AM)
|
||||
enabled_toolsets: ["web", "terminal"] # for fetching data
|
||||
```
|
||||
|
||||
### Cron Schedule Cheatsheet
|
||||
|
||||
| Pattern | Meaning |
|
||||
|---------|---------|
|
||||
| `30 8 * * 1-5` | Mon–Fri at 8:30 AM |
|
||||
| `0 8 * * 1` | Monday at 8 AM |
|
||||
| `0 9 * * *` | Every day at 9 AM |
|
||||
| `0 9 1 * *` | 1st of every month at 9 AM |
|
||||
|
||||
### Prompt Structure in the Cron Job
|
||||
|
||||
The prompt should be self-contained (cron jobs have no conversation context). Include:
|
||||
|
||||
1. **The agent's role** — "You are a [weather/stock/market] briefing agent"
|
||||
2. **Explicit data sources to call** — e.g. "curl -s wttr.in/Knoxville+TN+37919 to get the JSON forecast, then parse hourly data"
|
||||
3. **User constraints** — "Weekdays only after 6pm, weekends anytime"
|
||||
4. **Format instructions** — "Output as a Telegram-friendly message with emoji headers"
|
||||
5. **Anti-hallucination rule** — "Actually call the sources. Do not make up data. If a source fails, note it and move on."
|
||||
|
||||
## Step 3 — Data Sources
|
||||
|
||||
> **Firecrawl setup:** see `references/firecrawl-setup.md` for installation, API key config, and SDK usage patterns for scraping JS-heavy financial sites.
|
||||
|
||||
### Weather (free, no API key)
|
||||
|
||||
```bash
|
||||
# JSON forecast (3 days)
|
||||
curl -s "wttr.in/LOCATION?format=j1&days=3"
|
||||
|
||||
# Formatted terminal output
|
||||
curl -s "wttr.in/LOCATION?0&days=3"
|
||||
```
|
||||
|
||||
Parse hourly data to find dry windows matching the user's time constraints (after 6pm weekdays, anytime weekends). Key fields: `chanceofrain`, `precipMM`, `tempF`, `windspeedMiles`, `weatherDesc`.
|
||||
|
||||
### Stock/Financial Buzz
|
||||
|
||||
- **Firecrawl (preferred)** — see `references/firecrawl-setup.md` for setup. Bypasses JS-rendered paywalls and blocks that kill raw curl on Yahoo Finance, MarketWatch, etc. Free tier: 500 credits/month (~4 sources × 21 weekdays = 84 credits). **Firecrawl explicitly blocks Reddit** — do not attempt Reddit URLs with it.
|
||||
- **Yahoo Finance quote API** — `query1.finance.yahoo.com/v8/finance/chart/SYMBOL` still works with a proper User-Agent header. Use for real-time prices on discovered tickers.
|
||||
- **Google News RSS** — `news.google.com/rss/search?q=stock+market+today` works reliably via curl.
|
||||
- **Reddit (r/wallstreetbets only)** — the `.rss` endpoint at `https://www.reddit.com/r/wallstreetbets/.rss` returns valid Atom XML (HTTP 200). This is the ONLY working Reddit access path. Full Reddit scraping landscape: `references/reddit-scraping.md`. All other subs (r/stocks, r/StockMarket, r/pennystocks) return 403 on both `.rss` and `.json`. Parse XML titles for ticker mentions ($TICKER or all-caps symbols).
|
||||
|
||||
## Formatting for Telegram Delivery
|
||||
|
||||
Cron jobs auto-deliver to the home channel. Format for Telegram:
|
||||
|
||||
```
|
||||
📈 **Morning Stock Buzz — [Date]**
|
||||
|
||||
**🔥 Most Talked-About Stocks**
|
||||
- **TICKER** — why it's trending
|
||||
|
||||
**📰 Top Headlines**
|
||||
- Headline 1
|
||||
- Headline 2
|
||||
|
||||
**📊 Market Snapshot**
|
||||
- Futures direction, sector trends, notable movers
|
||||
```
|
||||
|
||||
Use: **bold**, bullet lists, emoji headers. No tables (Telegram rewrites them to bullets anyway).
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Self-contained prompts** — cron jobs have no memory of past conversations. Every detail the agent needs must be in the prompt.
|
||||
- **Rate-limited APIs** — GitHub and Reddit may 429 without proper auth. Use `GITHUB_TOKEN` env var for GitHub API calls.
|
||||
- **CGNAT blocks external access** — if a briefing references a self-hosted service (like Immich), the link must use the local LAN IP, not the CGNAT address.
|
||||
- **Reddit is perma-blocked** — the JSON API (`old.reddit.com/r/*.json`, `reddit.com/r/*/hot.json`) returns HTML blocks, not JSON. Firecrawl explicitly rejects Reddit (`403: "We do not support this site"`). The only working fallback: `r/wallstreetbets/.rss` returns valid Atom XML. All other subreddit RSS feeds (`r/stocks`, `r/StockMarket`, `r/pennystocks`) return HTTP 403. Use WSB `.rss` as the sole Reddit source in briefing prompts; skip Reddit entirely if WSB isn't sufficient.
|
||||
- **Firecrawl API key setup** — must be added directly to `~/.hermes/.env` (terminal echo/append), NOT via `hermes config set env.FIRECRAWL_API_KEY` which rejects the dotted key format. See `references/firecrawl-setup.md`.
|
||||
- **Weekend vs weekday** — stock/market briefings should run Mon–Fri only ($30 8 * * 1-5$). Weather/mowing briefings can include weekends.
|
||||
- **No `web_search` tool** — use `terminal` with `curl` instead. Design the prompt to use curl-based data collection.
|
||||
|
||||
## Alternative: no_agent=True (Watchdog Pattern)
|
||||
|
||||
For **deterministic monitoring** where the output is pre-defined (drive health checks, service uptime, disk space alerts), use `no_agent=True` with a script instead of the LLM-driven approach:
|
||||
|
||||
```python
|
||||
cronjob(
|
||||
action="create",
|
||||
name="daily-disk-check",
|
||||
schedule="0 5 * * *",
|
||||
script="disk-alert.py", # ~/.hermes/scripts/disk-alert.py
|
||||
no_agent=True # script stdout = delivery message
|
||||
)
|
||||
```
|
||||
|
||||
**When to use `no_agent=True` vs default (LLM-driven):**
|
||||
|
||||
| Aspect | `no_agent=True` (watchdog) | `no_agent=False` (briefing) |
|
||||
|--------|---------------------------|---------------------------|
|
||||
| Cost | Zero tokens per run | LLM inference per run |
|
||||
| Output | Script stdout verbatim | Agent formats the message |
|
||||
| Use case | Threshold checks, alerts, pings | Summaries, digests, formatted briefings |
|
||||
| Silent on no-op | Script prints nothing = silent | Agent always produces output |
|
||||
|
||||
**Key behaviors:**
|
||||
- Non-empty stdin → delivered as the message
|
||||
- Empty stdout → SILENT (nothing sent to user — perfect for "all clear" checks)
|
||||
- Non-zero exit / timeout → error alert sent to user (can't fail silently)
|
||||
@@ -0,0 +1,54 @@
|
||||
# Example Briefing Configurations
|
||||
|
||||
Concrete cron job setups from a real session. Use these as templates when creating new briefings.
|
||||
|
||||
## Example 1: Weather / Mowing Forecast
|
||||
|
||||
**Trigger:** "Check weather in Knoxville, TN 37919 — best mowing day after 6pm weekdays, anytime weekends"
|
||||
|
||||
**Cron settings:**
|
||||
```
|
||||
schedule: 0 8 * * 1 (Monday 8 AM)
|
||||
toolsets: ["web", "terminal"]
|
||||
deliver: origin (sends to this chat)
|
||||
```
|
||||
|
||||
**Prompt structure (simplified):**
|
||||
```
|
||||
Fetch the JSON forecast from wttr.in for Knoxville TN 37919.
|
||||
Parse hourly data for each available day.
|
||||
Check constraints: after 6pm on weekdays, anytime on weekends.
|
||||
Find the slot with lowest rain chance, comfortable temp, low wind.
|
||||
Format with 🌱 emoji, bold the recommendation, keep it concise.
|
||||
```
|
||||
|
||||
## Example 2: Stock Buzz Briefing
|
||||
|
||||
**Trigger:** "Most talked-about stocks each morning from Reddit, news, and general buzz"
|
||||
|
||||
**Cron settings:**
|
||||
```
|
||||
schedule: 30 8 * * 1-5 (Mon-Fri 8:30 AM)
|
||||
toolsets: ["web", "terminal"]
|
||||
deliver: origin
|
||||
```
|
||||
|
||||
**Data sources to scrape:**
|
||||
- Reddit: `old.reddit.com/r/wallstreetbets.json`, `r/stocks.json`, `r/investing.json`, `r/StockMarket.json`
|
||||
- News: `finance.yahoo.com`, market aggregator sites
|
||||
- Pre-market movers from free financial sources
|
||||
|
||||
**Prompt structure (simplified):**
|
||||
```
|
||||
Gather trending tickers from Reddit finance subs, financial news headlines, and general buzz.
|
||||
Categorize: most talked-about stocks, top headlines, Reddit pulse.
|
||||
Format with 📈 emoji header, bold tickers, keep to 3-5 stocks unless event-driven.
|
||||
```
|
||||
|
||||
## Common cron schedule patterns used
|
||||
|
||||
| Pattern | Meaning | Use Case |
|
||||
|---------|---------|----------|
|
||||
| `30 8 * * 1-5` | 8:30 AM weekdays | Stock buzz (markets open) |
|
||||
| `0 8 * * 1` | 8 AM Mondays | Weekly weather/mowing |
|
||||
| `0 9 * * *` | 9 AM daily | Daily digests |
|
||||
@@ -0,0 +1,88 @@
|
||||
# Firecrawl Setup for Cron Jobs
|
||||
|
||||
Firecrawl is a web scraping API that returns clean markdown from any URL — bypasses JS-rendered blocks, Cloudflare, and paywalls that kill raw curl.
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
pip install firecrawl-py
|
||||
```
|
||||
|
||||
## API Key Setup
|
||||
|
||||
**Critical:** Must be added directly to `~/.hermes/.env`, NOT via `hermes config set`:
|
||||
|
||||
```bash
|
||||
# WRONG — this fails with "Invalid environment variable name":
|
||||
hermes config set env.FIRECRAWL_API_KEY "fc-..."
|
||||
|
||||
# CORRECT — append directly:
|
||||
echo 'FIRECRAWL_API_KEY=fc-...' >> ~/.hermes/.env
|
||||
```
|
||||
|
||||
The `hermes config set env.KEY value` command rejects dotted keys. Use terminal append to `.env` instead.
|
||||
|
||||
Verify it's set:
|
||||
```bash
|
||||
grep FIRECRAWL ~/.hermes/.env
|
||||
```
|
||||
|
||||
⚠️ **Pitfall:** if you appended the key but a commented-out `# FIRECRAWL_API_KEY=...` line exists earlier in the file, the first (commented) match will be picked up by scripts that iterate line-by-line. Always use `not line.strip().startswith('#')` when parsing `.env` programmatically, or delete the commented line.
|
||||
|
||||
## SDK Usage Pattern
|
||||
|
||||
Use this in cron job prompts or agent scripts:
|
||||
|
||||
```python
|
||||
from firecrawl import FirecrawlApp
|
||||
import os
|
||||
|
||||
app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
|
||||
|
||||
# Scrape a URL — returns {'markdown': '...', 'metadata': {...}}
|
||||
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
|
||||
print(result['markdown'][:3000]) # trim for context window
|
||||
```
|
||||
|
||||
### Key API details
|
||||
- Method: `app.scrape_url(url, formats=['markdown'])` — NOT `params=` keyword
|
||||
- Returns: dict with `markdown` (cleaned text), `metadata` (title, description, etc.)
|
||||
- Each scrape = 1 credit on the free tier (500/month)
|
||||
|
||||
## Free Tier
|
||||
|
||||
- 500 credits/month
|
||||
- ~84 credits/month for a weekday stock briefing (4 sources × ~21 weekdays)
|
||||
- Well within limits even with occasional retries
|
||||
|
||||
## Common Sources That Work
|
||||
|
||||
| Source | URL | Notes |
|
||||
|--------|-----|-------|
|
||||
| Yahoo Finance | `https://finance.yahoo.com/` | JS-heavy; curl fails, Firecrawl works |
|
||||
| MarketWatch | `https://www.marketwatch.com/` | Paywall-aware; returns article text |
|
||||
| Google News | `https://news.google.com/search?q=...` | Also works via RSS |
|
||||
| Finviz | `https://finviz.com/` | Market heatmaps, gainers/losers |
|
||||
|
||||
## Sources That Do NOT Work
|
||||
|
||||
| Source | Method | Error |
|
||||
|--------|--------|-------|
|
||||
| **Reddit (any sub)** | Firecrawl | `403: "We do not support this site"` |
|
||||
| Reddit JSON API | `curl old.reddit.com/r/*/hot.json` | Returns HTML "whoa there, pardner" block |
|
||||
| r/stocks RSS | `curl reddit.com/r/stocks/.rss` | `HTTP 403` |
|
||||
| r/StockMarket RSS | `curl reddit.com/r/StockMarket/.rss` | `HTTP 403` |
|
||||
| r/pennystocks RSS | `curl reddit.com/r/pennystocks/.rss` | `HTTP 403` |
|
||||
|
||||
**Only working Reddit access:** `r/wallstreetbets/.rss` returns valid Atom XML (HTTP 200). All other subreddit RSS feeds and the JSON API are blocked. For stock briefings, scrape WSB `.rss` as the sole Reddit source and skip the rest.
|
||||
|
||||
## Fallback: Yahoo Finance Quote API
|
||||
|
||||
For real-time prices without Firecrawl (still works with curl):
|
||||
|
||||
```bash
|
||||
curl -s "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=1d&interval=5m" \
|
||||
-H "User-Agent: Mozilla/5.0"
|
||||
```
|
||||
|
||||
Parse with: `data['chart']['result'][0]['meta']['regularMarketPrice']`
|
||||
@@ -0,0 +1,42 @@
|
||||
# Reddit Scraping for Briefings
|
||||
|
||||
Reddit aggressively blocks programmatic access. Here's the definitive landscape as of June 2026.
|
||||
|
||||
## What Works
|
||||
|
||||
| Method | Subreddit | Status |
|
||||
|--------|-----------|--------|
|
||||
| `.rss` (Atom XML) | `r/wallstreetbets` | ✅ **HTTP 200** — valid XML with titles |
|
||||
| `.rss` (Atom XML) | All other subs | ❌ HTTP 403 |
|
||||
|
||||
**That's it.** Only WSB's RSS feed is accessible without authentication.
|
||||
|
||||
## What Does NOT Work
|
||||
|
||||
| Method | Subreddit | Error |
|
||||
|--------|-----------|-------|
|
||||
| Firecrawl SDK | Any Reddit URL | `403: "We do not support this site"` |
|
||||
| `.json` API | Any sub | Returns HTML "whoa there, pardner" |
|
||||
| `.rss` | `r/stocks` | `HTTP 403` |
|
||||
| `.rss` | `r/StockMarket` | `HTTP 403` |
|
||||
| `.rss` | `r/pennystocks` | `HTTP 403` |
|
||||
| `old.reddit.com` | Any sub | Same blocks as `www.reddit.com` |
|
||||
|
||||
## WSB RSS Pattern
|
||||
|
||||
```bash
|
||||
curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
|
||||
"https://www.reddit.com/r/wallstreetbets/.rss" -o /tmp/wsb.xml
|
||||
```
|
||||
|
||||
Parse the Atom XML for `<title>` elements. Extract ticker mentions (`$TICKER` or ASYMBOL patterns) and sentiment from titles.
|
||||
|
||||
## If You Need More Reddit Data
|
||||
|
||||
- **Reddit's official API** requires OAuth and app registration at reddit.com/prefs/apps. Even then, rate limits are strict for free-tier apps.
|
||||
- **Pushshift** (historical Reddit data) has been unreliable since 2023 API changes.
|
||||
- **Paid alternatives**: SocialGrep, TrackReddit, etc. — none tested from cron.
|
||||
|
||||
## Recommendation
|
||||
|
||||
For stock briefing cron jobs: use WSB `.rss` as the sole Reddit source. Do not attempt to scrape other subreddits — they will fail every time. If WSB chatter isn't enough, supplement with StockTwits trending or financial news sentiment instead.
|
||||
Reference in New Issue
Block a user