ray
89 lines
3.2 KiB
Markdown
89 lines
3.2 KiB
Markdown
# Firecrawl Setup for Cron Jobs
|
||
|
||
Firecrawl is a web scraping API that returns clean markdown from any URL — bypasses JS-rendered blocks, Cloudflare, and paywalls that kill raw curl.
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
pip install firecrawl-py
|
||
```
|
||
|
||
## API Key Setup
|
||
|
||
**Critical:** Must be added directly to `~/.hermes/.env`, NOT via `hermes config set`:
|
||
|
||
```bash
|
||
# WRONG — this fails with "Invalid environment variable name":
|
||
hermes config set env.FIRECRAWL_API_KEY "fc-..."
|
||
|
||
# CORRECT — append directly:
|
||
echo 'FIRECRAWL_API_KEY=fc-...' >> ~/.hermes/.env
|
||
```
|
||
|
||
The `hermes config set env.KEY value` command rejects dotted keys. Use terminal append to `.env` instead.
|
||
|
||
Verify it's set:
|
||
```bash
|
||
grep FIRECRAWL ~/.hermes/.env
|
||
```
|
||
|
||
⚠️ **Pitfall:** if you appended the key but a commented-out `# FIRECRAWL_API_KEY=...` line exists earlier in the file, the first (commented) match will be picked up by scripts that iterate line-by-line. Always use `not line.strip().startswith('#')` when parsing `.env` programmatically, or delete the commented line.
|
||
|
||
## SDK Usage Pattern
|
||
|
||
Use this in cron job prompts or agent scripts:
|
||
|
||
```python
|
||
from firecrawl import FirecrawlApp
|
||
import os
|
||
|
||
app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
|
||
|
||
# Scrape a URL — returns {'markdown': '...', 'metadata': {...}}
|
||
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
|
||
print(result['markdown'][:3000]) # trim for context window
|
||
```
|
||
|
||
### Key API details
|
||
- Method: `app.scrape_url(url, formats=['markdown'])` — NOT `params=` keyword
|
||
- Returns: dict with `markdown` (cleaned text), `metadata` (title, description, etc.)
|
||
- Each scrape = 1 credit on the free tier (500/month)
|
||
|
||
## Free Tier
|
||
|
||
- 500 credits/month
|
||
- ~84 credits/month for a weekday stock briefing (4 sources × ~21 weekdays)
|
||
- Well within limits even with occasional retries
|
||
|
||
## Common Sources That Work
|
||
|
||
| Source | URL | Notes |
|
||
|--------|-----|-------|
|
||
| Yahoo Finance | `https://finance.yahoo.com/` | JS-heavy; curl fails, Firecrawl works |
|
||
| MarketWatch | `https://www.marketwatch.com/` | Paywall-aware; returns article text |
|
||
| Google News | `https://news.google.com/search?q=...` | Also works via RSS |
|
||
| Finviz | `https://finviz.com/` | Market heatmaps, gainers/losers |
|
||
|
||
## Sources That Do NOT Work
|
||
|
||
| Source | Method | Error |
|
||
|--------|--------|-------|
|
||
| **Reddit (any sub)** | Firecrawl | `403: "We do not support this site"` |
|
||
| Reddit JSON API | `curl old.reddit.com/r/*/hot.json` | Returns HTML "whoa there, pardner" block |
|
||
| r/stocks RSS | `curl reddit.com/r/stocks/.rss` | `HTTP 403` |
|
||
| r/StockMarket RSS | `curl reddit.com/r/StockMarket/.rss` | `HTTP 403` |
|
||
| r/pennystocks RSS | `curl reddit.com/r/pennystocks/.rss` | `HTTP 403` |
|
||
|
||
**Only working Reddit access:** `r/wallstreetbets/.rss` returns valid Atom XML (HTTP 200). All other subreddit RSS feeds and the JSON API are blocked. For stock briefings, scrape WSB `.rss` as the sole Reddit source and skip the rest.
|
||
|
||
## Fallback: Yahoo Finance Quote API
|
||
|
||
For real-time prices without Firecrawl (still works with curl):
|
||
|
||
```bash
|
||
curl -s "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?range=1d&interval=5m" \
|
||
-H "User-Agent: Mozilla/5.0"
|
||
```
|
||
|
||
Parse with: `data['chart']['result'][0]['meta']['regularMarketPrice']`
|