initial commit

2026-07-12 10:17:17 -04:00
commit dab5a4ebc6
1424 changed files with 330463 additions and 0 deletions
@@ -0,0 +1,42 @@
+# Reddit Scraping for Briefings
+
+Reddit aggressively blocks programmatic access. Here's the definitive landscape as of June 2026.
+
+## What Works
+
+| Method | Subreddit | Status |
+|--------|-----------|--------|
+| `.rss` (Atom XML) | `r/wallstreetbets` | ✅ **HTTP 200** — valid XML with titles |
+| `.rss` (Atom XML) | All other subs | ❌ HTTP 403 |
+
+**That's it.** Only WSB's RSS feed is accessible without authentication.
+
+## What Does NOT Work
+
+| Method | Subreddit | Error |
+|--------|-----------|-------|
+| Firecrawl SDK | Any Reddit URL | `403: "We do not support this site"` |
+| `.json` API | Any sub | Returns HTML "whoa there, pardner" |
+| `.rss` | `r/stocks` | `HTTP 403` |
+| `.rss` | `r/StockMarket` | `HTTP 403` |
+| `.rss` | `r/pennystocks` | `HTTP 403` |
+| `old.reddit.com` | Any sub | Same blocks as `www.reddit.com` |
+
+## WSB RSS Pattern
+
+```bash
+curl -sL -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64)" \
+  "https://www.reddit.com/r/wallstreetbets/.rss" -o /tmp/wsb.xml
+```
+
+Parse the Atom XML for `<title>` elements. Extract ticker mentions (`$TICKER` or ASYMBOL patterns) and sentiment from titles.
+
+## If You Need More Reddit Data
+
+- **Reddit's official API** requires OAuth and app registration at reddit.com/prefs/apps. Even then, rate limits are strict for free-tier apps.
+- **Pushshift** (historical Reddit data) has been unreliable since 2023 API changes.
+- **Paid alternatives**: SocialGrep, TrackReddit, etc. — none tested from cron.
+
+## Recommendation
+
+For stock briefing cron jobs: use WSB `.rss` as the sole Reddit source. Do not attempt to scrape other subreddits — they will fail every time. If WSB chatter isn't enough, supplement with StockTwits trending or financial news sentiment instead.