ray
43 lines
1.7 KiB
Markdown
43 lines
1.7 KiB
Markdown
# Reddit Scraping for Briefings
|
|
|
|
Reddit aggressively blocks programmatic access. Here's the definitive landscape as of June 2026.
|
|
|
|
## What Works
|
|
|
|
| Method | Subreddit | Status |
|
|
|--------|-----------|--------|
|
|
| `.rss` (Atom XML) | `r/wallstreetbets` | ✅ **HTTP 200** — valid XML with titles |
|
|
| `.rss` (Atom XML) | All other subs | ❌ HTTP 403 |
|
|
|
|
**That's it.** Only WSB's RSS feed is accessible without authentication.
|
|
|
|
## What Does NOT Work
|
|
|
|
| Method | Subreddit | Error |
|
|
|--------|-----------|-------|
|
|
| Firecrawl SDK | Any Reddit URL | `403: "We do not support this site"` |
|
|
| `.json` API | Any sub | Returns HTML "whoa there, pardner" |
|
|
| `.rss` | `r/stocks` | `HTTP 403` |
|
|
| `.rss` | `r/StockMarket` | `HTTP 403` |
|
|
| `.rss` | `r/pennystocks` | `HTTP 403` |
|
|
| `old.reddit.com` | Any sub | Same blocks as `www.reddit.com` |
|
|
|
|
## WSB RSS Pattern
|
|
|
|
```bash
|
|
curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
|
|
"https://www.reddit.com/r/wallstreetbets/.rss" -o /tmp/wsb.xml
|
|
```
|
|
|
|
Parse the Atom XML for `<title>` elements. Extract ticker mentions (`$TICKER` or ASYMBOL patterns) and sentiment from titles.
|
|
|
|
## If You Need More Reddit Data
|
|
|
|
- **Reddit's official API** requires OAuth and app registration at reddit.com/prefs/apps. Even then, rate limits are strict for free-tier apps.
|
|
- **Pushshift** (historical Reddit data) has been unreliable since 2023 API changes.
|
|
- **Paid alternatives**: SocialGrep, TrackReddit, etc. — none tested from cron.
|
|
|
|
## Recommendation
|
|
|
|
For stock briefing cron jobs: use WSB `.rss` as the sole Reddit source. Do not attempt to scrape other subreddits — they will fail every time. If WSB chatter isn't enough, supplement with StockTwits trending or financial news sentiment instead.
|