# Firecrawl + Reddit Limitations

## Discovery (June 2026)

Firecrawl (firecrawl.dev, `firecrawl-py` SDK) **explicitly blocks Reddit domains**:

```
HTTP 403: "We apologize for the inconvenience but we do not support this site."
```

This applies to all Reddit URLs: `old.reddit.com`, `www.reddit.com`, `reddit.com`, and all subreddits.

## Workaround: Reddit RSS Feeds

Reddit provides Atom/RSS feeds at `https://www.reddit.com/r/<subreddit>/.rss`. These are less aggressively blocked than the JSON API.

**Availability (tested June 2026):**

| Subreddit | `.rss` | `.json` API |
|-----------|--------|-------------|
| r/wallstreetbets | ✅ 200 | ❌ Blocked |
| r/stocks | ❌ 403 | ❌ Blocked |
| r/StockMarket | ❌ 403 | ❌ Blocked |
| r/pennystocks | ❌ 403 | ❌ Blocked |

Only r/wallstreetbets RSS is accessible. Other financial subreddits block all programmatic access.

## Fetching WSB RSS

```bash
curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
  "https://www.reddit.com/r/wallstreetbets/.rss"
```

Returns Atom XML with post titles, links, and timestamps. Parse for `$TICKER` mentions and sentiment.

## What Doesn't Work

- Firecrawl SDK on any Reddit URL
- Reddit `.json` API (returns HTML blocks: "whoa there, pardner")
- `old.reddit.com/r/*/.json` (same block)
- `api.reddit.com/r/*/hot` (same block)
- Most subreddit RSS feeds (403 on non-WSB subs)

## Firecrawl Works For

Financial news sites are fine:
- `finance.yahoo.com` ✅
- `marketwatch.com` ✅
- `finviz.com` ✅
- `news.google.com` ✅

## Python SDK Usage

```python
from firecrawl import FirecrawlApp
import os

app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])

# Correct: formats is a keyword argument (list), NOT params=dict
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
md = result.get('markdown', '')
```

Install: `pip install firecrawl-py` (v4.28.2 tested). Free tier: 500 credits/month, 1 credit per scrape.

API key format: `fc-1a44...` (35 chars). Store in `~/.hermes/.env` as `FIRECRAWL_API_KEY`. When reading from `.env`, skip commented lines (grep for lines NOT starting with `#`).