ray
69 lines
2.1 KiB
Markdown
69 lines
2.1 KiB
Markdown
# Firecrawl + Reddit Limitations
|
|
|
|
## Discovery (June 2026)
|
|
|
|
Firecrawl (firecrawl.dev, `firecrawl-py` SDK) **explicitly blocks Reddit domains**:
|
|
|
|
```
|
|
HTTP 403: "We apologize for the inconvenience but we do not support this site."
|
|
```
|
|
|
|
This applies to all Reddit URLs: `old.reddit.com`, `www.reddit.com`, `reddit.com`, and all subreddits.
|
|
|
|
## Workaround: Reddit RSS Feeds
|
|
|
|
Reddit provides Atom/RSS feeds at `https://www.reddit.com/r/<subreddit>/.rss`. These are less aggressively blocked than the JSON API.
|
|
|
|
**Availability (tested June 2026):**
|
|
|
|
| Subreddit | `.rss` | `.json` API |
|
|
|-----------|--------|-------------|
|
|
| r/wallstreetbets | ✅ 200 | ❌ Blocked |
|
|
| r/stocks | ❌ 403 | ❌ Blocked |
|
|
| r/StockMarket | ❌ 403 | ❌ Blocked |
|
|
| r/pennystocks | ❌ 403 | ❌ Blocked |
|
|
|
|
Only r/wallstreetbets RSS is accessible. Other financial subreddits block all programmatic access.
|
|
|
|
## Fetching WSB RSS
|
|
|
|
```bash
|
|
curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
|
|
"https://www.reddit.com/r/wallstreetbets/.rss"
|
|
```
|
|
|
|
Returns Atom XML with post titles, links, and timestamps. Parse for `$TICKER` mentions and sentiment.
|
|
|
|
## What Doesn't Work
|
|
|
|
- Firecrawl SDK on any Reddit URL
|
|
- Reddit `.json` API (returns HTML blocks: "whoa there, pardner")
|
|
- `old.reddit.com/r/*/.json` (same block)
|
|
- `api.reddit.com/r/*/hot` (same block)
|
|
- Most subreddit RSS feeds (403 on non-WSB subs)
|
|
|
|
## Firecrawl Works For
|
|
|
|
Financial news sites are fine:
|
|
- `finance.yahoo.com` ✅
|
|
- `marketwatch.com` ✅
|
|
- `finviz.com` ✅
|
|
- `news.google.com` ✅
|
|
|
|
## Python SDK Usage
|
|
|
|
```python
|
|
from firecrawl import FirecrawlApp
|
|
import os
|
|
|
|
app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])
|
|
|
|
# Correct: formats is a keyword argument (list), NOT params=dict
|
|
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
|
|
md = result.get('markdown', '')
|
|
```
|
|
|
|
Install: `pip install firecrawl-py` (v4.28.2 tested). Free tier: 500 credits/month, 1 credit per scrape.
|
|
|
|
API key format: `fc-1a44...` (35 chars). Store in `~/.hermes/.env` as `FIRECRAWL_API_KEY`. When reading from `.env`, skip commented lines (grep for lines NOT starting with `#`).
|