Firecrawl Setup for Cron Jobs

Firecrawl is a web scraping API that returns clean markdown from any URL — bypasses JS-rendered blocks, Cloudflare, and paywalls that kill raw curl.

Installation

pip install firecrawl-py

API Key Setup

Critical: Must be added directly to ~/.hermes/.env, NOT via hermes config set:

# WRONG — this fails with "Invalid environment variable name":
hermes config set env.FIRECRAWL_API_KEY "fc-..."

# CORRECT — append directly:
echo 'FIRECRAWL_API_KEY=fc-...' >> ~/.hermes/.env

The hermes config set env.KEY value command rejects dotted keys. Use terminal append to .env instead.

Verify it's set:

grep FIRECRAWL ~/.hermes/.env

⚠️ Pitfall: if you appended the key but a commented-out # FIRECRAWL_API_KEY=... line exists earlier in the file, the first (commented) match will be picked up by scripts that iterate line-by-line. Always use not line.strip().startswith('#') when parsing .env programmatically, or delete the commented line.

SDK Usage Pattern

Use this in cron job prompts or agent scripts:

from firecrawl import FirecrawlApp
import os

app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])

# Scrape a URL — returns {'markdown': '...', 'metadata': {...}}
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
print(result['markdown'][:3000])  # trim for context window

Key API details

Method: app.scrape_url(url, formats=['markdown']) — NOT params= keyword
Returns: dict with markdown (cleaned text), metadata (title, description, etc.)
Each scrape = 1 credit on the free tier (500/month)

Free Tier

500 credits/month
~84 credits/month for a weekday stock briefing (4 sources × ~21 weekdays)
Well within limits even with occasional retries

Common Sources That Work

Source	URL	Notes
Yahoo Finance	`https://finance.yahoo.com/`	JS-heavy; curl fails, Firecrawl works
MarketWatch	`https://www.marketwatch.com/`	Paywall-aware; returns article text
Google News	`https://news.google.com/search?q=...`	Also works via RSS
Finviz	`https://finviz.com/`	Market heatmaps, gainers/losers

Sources That Do NOT Work

Source	Method	Error
Reddit (any sub)	Firecrawl	`403: "We do not support this site"`
Reddit JSON API	`curl old.reddit.com/r/*/hot.json`	Returns HTML "whoa there, pardner" block
r/stocks RSS	`curl reddit.com/r/stocks/.rss`	`HTTP 403`
r/StockMarket RSS	`curl reddit.com/r/StockMarket/.rss`	`HTTP 403`
r/pennystocks RSS	`curl reddit.com/r/pennystocks/.rss`	`HTTP 403`

Only working Reddit access: r/wallstreetbets/.rss returns valid Atom XML (HTTP 200). All other subreddit RSS feeds and the JSON API are blocked. For stock briefings, scrape WSB .rss as the sole Reddit source and skip the rest.

Fallback: Yahoo Finance Quote API

For real-time prices without Firecrawl (still works with curl):

curl -s "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=1d&interval=5m" \
  -H "User-Agent: Mozilla/5.0"

Parse with: data['chart']['result'][0]['meta']['regularMarketPrice']

3.2 KiB Raw Permalink Blame History Unescape Escape