Firecrawl + Reddit Limitations

Discovery (June 2026)

Firecrawl (firecrawl.dev, firecrawl-py SDK) explicitly blocks Reddit domains:

HTTP 403: "We apologize for the inconvenience but we do not support this site."

This applies to all Reddit URLs: old.reddit.com, www.reddit.com, reddit.com, and all subreddits.

Workaround: Reddit RSS Feeds

Reddit provides Atom/RSS feeds at https://www.reddit.com/r/<subreddit>/.rss. These are less aggressively blocked than the JSON API.

Availability (tested June 2026):

Subreddit	`.rss`	`.json` API
r/wallstreetbets	✅ 200	❌ Blocked
r/stocks	❌ 403	❌ Blocked
r/StockMarket	❌ 403	❌ Blocked
r/pennystocks	❌ 403	❌ Blocked

Only r/wallstreetbets RSS is accessible. Other financial subreddits block all programmatic access.

Fetching WSB RSS

curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
  "https://www.reddit.com/r/wallstreetbets/.rss"

Returns Atom XML with post titles, links, and timestamps. Parse for $TICKER mentions and sentiment.

What Doesn't Work

Firecrawl SDK on any Reddit URL
Reddit .json API (returns HTML blocks: "whoa there, pardner")
old.reddit.com/r/*/.json (same block)
api.reddit.com/r/*/hot (same block)
Most subreddit RSS feeds (403 on non-WSB subs)

Firecrawl Works For

Financial news sites are fine:

finance.yahoo.com ✅
marketwatch.com ✅
finviz.com ✅
news.google.com ✅

Python SDK Usage

from firecrawl import FirecrawlApp
import os

app = FirecrawlApp(api_key=os.environ['FIRECRAWL_API_KEY'])

# Correct: formats is a keyword argument (list), NOT params=dict
result = app.scrape_url('https://finance.yahoo.com/', formats=['markdown'])
md = result.get('markdown', '')

Install: pip install firecrawl-py (v4.28.2 tested). Free tier: 500 credits/month, 1 credit per scrape.

API key format: fc-1a44... (35 chars). Store in ~/.hermes/.env as FIRECRAWL_API_KEY. When reading from .env, skip commented lines (grep for lines NOT starting with #).

2.1 KiB Raw Blame History