initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,54 @@
+# Example Briefing Configurations
+
+Concrete cron job setups from a real session. Use these as templates when creating new briefings.
+
+## Example 1: Weather / Mowing Forecast
+
+**Trigger:** "Check weather in Knoxville, TN 37919 — best mowing day after 6pm weekdays, anytime weekends"
+
+**Cron settings:**
+```
+schedule: 0 8 * * 1      (Monday 8 AM)
+toolsets: ["web", "terminal"]
+deliver: origin            (sends to this chat)
+```
+
+**Prompt structure (simplified):**
+```
+Fetch the JSON forecast from wttr.in for Knoxville TN 37919.
+Parse hourly data for each available day.
+Check constraints: after 6pm on weekdays, anytime on weekends.
+Find the slot with lowest rain chance, comfortable temp, low wind.
+Format with 🌱 emoji, bold the recommendation, keep it concise.
+```
+
+## Example 2: Stock Buzz Briefing
+
+**Trigger:** "Most talked-about stocks each morning from Reddit, news, and general buzz"
+
+**Cron settings:**
+```
+schedule: 30 8 * * 1-5    (Mon-Fri 8:30 AM)
+toolsets: ["web", "terminal"]
+deliver: origin
+```
+
+**Data sources to scrape:**
+- Reddit: `old.reddit.com/r/wallstreetbets.json`, `r/stocks.json`, `r/investing.json`, `r/StockMarket.json`
+- News: `finance.yahoo.com`, market aggregator sites
+- Pre-market movers from free financial sources
+
+**Prompt structure (simplified):**
+```
+Gather trending tickers from Reddit finance subs, financial news headlines, and general buzz.
+Categorize: most talked-about stocks, top headlines, Reddit pulse.
+Format with 📈 emoji header, bold tickers, keep to 3-5 stocks unless event-driven.
+```
+
+## Common cron schedule patterns used
+
+| Pattern | Meaning | Use Case |
+|---------|---------|----------|
+| `30 8 * * 1-5` | 8:30 AM weekdays | Stock buzz (markets open) |
+| `0 8 * * 1` | 8 AM Mondays | Weekly weather/mowing |
+| `0 9 * * *` | 9 AM daily | Daily digests |
@@ -0,0 +1,88 @@
+# Firecrawl Setup for Cron Jobs
+
+Firecrawl is a web scraping API that returns clean markdown from any URL — bypasses JS-rendered blocks, Cloudflare, and paywalls that kill raw curl.
+
+## Installation
+
+```bash
+pip install firecrawl-py
+```
+
+## API Key Setup
+
+**Critical:** Must be added directly to `~/.hermes/.env`, NOT via `hermes config set`:
+
+```bash
+# WRONG — this fails with "Invalid environment variable name":
+hermes config set env.FIRECRAWL_API_KEY "fc-..."
+
+# CORRECT — append directly:
+echo 'FIRECRAWL_API_KEY=fc-...' >> ~/.hermes/.env
+```
+
+The `hermes config set env.KEY value` command rejects dotted keys. Use terminal append to `.env` instead.
+
+Verify it's set:
+```bash
+grep FIRECRAWL ~/.hermes/.env
+```
+
+⚠️ **Pitfall:** if you appended the key but a commented-out `# FIRECRAWL_API_KEY=...` line exists earlier in the file, the first (commented) match will be picked up by scripts that iterate line-by-line. Always use `not line.strip().startswith('#')` when parsing `.env` programmatically, or delete the commented line.
+
+## SDK Usage Pattern
+
+Use this in cron job prompts or agent scripts:
+
+```python
+from firecrawl import FirecrawlApp
+import os
+
+app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
+
+# Scrape a URL — returns {'markdown': '...', 'metadata': {...}}
+result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
+print(result['markdown'][:3000])  # trim for context window
+```
+
+### Key API details
+- Method: `app.scrape_url(url, formats=['markdown'])` — NOT `params=` keyword
+- Returns: dict with `markdown` (cleaned text), `metadata` (title, description, etc.)
+- Each scrape = 1 credit on the free tier (500/month)
+
+## Free Tier
+
+- 500 credits/month
+- ~84 credits/month for a weekday stock briefing (4 sources × ~21 weekdays)
+- Well within limits even with occasional retries
+
+## Common Sources That Work
+
+| Source | URL | Notes |
+|--------|-----|-------|
+| Yahoo Finance | `https://finance.yahoo.com/` | JS-heavy; curl fails, Firecrawl works |
+| MarketWatch | `https://www.marketwatch.com/` | Paywall-aware; returns article text |
+| Google News | `https://news.google.com/search?q=...` | Also works via RSS |
+| Finviz | `https://finviz.com/` | Market heatmaps, gainers/losers |
+
+## Sources That Do NOT Work
+
+| Source | Method | Error |
+|--------|--------|-------|
+| **Reddit (any sub)** | Firecrawl | `403: "We do not support this site"` |
+| Reddit JSON API | `curl old.reddit.com/r/*/hot.json` | Returns HTML "whoa there, pardner" block |
+| r/stocks RSS | `curl reddit.com/r/stocks/.rss` | `HTTP 403` |
+| r/StockMarket RSS | `curl reddit.com/r/StockMarket/.rss` | `HTTP 403` |
+| r/pennystocks RSS | `curl reddit.com/r/pennystocks/.rss` | `HTTP 403` |
+
+**Only working Reddit access:** `r/wallstreetbets/.rss` returns valid Atom XML (HTTP 200). All other subreddit RSS feeds and the JSON API are blocked. For stock briefings, scrape WSB `.rss` as the sole Reddit source and skip the rest.
+
+## Fallback: Yahoo Finance Quote API
+
+For real-time prices without Firecrawl (still works with curl):
+
+```bash
+curl -s "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=1d&interval=5m" \
+  -H "User-Agent: Mozilla/5.0"
+```
+
+Parse with: `data['chart']['result'][0]['meta']['regularMarketPrice']`
@@ -0,0 +1,42 @@
+# Reddit Scraping for Briefings
+
+Reddit aggressively blocks programmatic access. Here's the definitive landscape as of June 2026.
+
+## What Works
+
+| Method | Subreddit | Status |
+|--------|-----------|--------|
+| `.rss` (Atom XML) | `r/wallstreetbets` | ✅ **HTTP 200** — valid XML with titles |
+| `.rss` (Atom XML) | All other subs | ❌ HTTP 403 |
+
+**That's it.** Only WSB's RSS feed is accessible without authentication.
+
+## What Does NOT Work
+
+| Method | Subreddit | Error |
+|--------|-----------|-------|
+| Firecrawl SDK | Any Reddit URL | `403: "We do not support this site"` |
+| `.json` API | Any sub | Returns HTML "whoa there, pardner" |
+| `.rss` | `r/stocks` | `HTTP 403` |
+| `.rss` | `r/StockMarket` | `HTTP 403` |
+| `.rss` | `r/pennystocks` | `HTTP 403` |
+| `old.reddit.com` | Any sub | Same blocks as `www.reddit.com` |
+
+## WSB RSS Pattern
+
+```bash
+curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
+  "https://www.reddit.com/r/wallstreetbets/.rss" -o /tmp/wsb.xml
+```
+
+Parse the Atom XML for `<title>` elements. Extract ticker mentions (`$TICKER` or ASYMBOL patterns) and sentiment from titles.
+
+## If You Need More Reddit Data
+
+- **Reddit's official API** requires OAuth and app registration at reddit.com/prefs/apps. Even then, rate limits are strict for free-tier apps.
+- **Pushshift** (historical Reddit data) has been unreliable since 2023 API changes.
+- **Paid alternatives**: SocialGrep, TrackReddit, etc. — none tested from cron.
+
+## Recommendation
+
+For stock briefing cron jobs: use WSB `.rss` as the sole Reddit source. Do not attempt to scrape other subreddits — they will fail every time. If WSB chatter isn't enough, supplement with StockTwits trending or financial news sentiment instead.