hot-news-crawler

---
name: "hot-news-crawler"
description: "Crawl and aggregate trending tech/AI news from multiple platforms. Tracks HackerNews, Lobsters, GitHub Trending, Product Hunt, TechMeme, NewsAPI headlines, plus AI/tech leaders' blogs and YouTube channels via RSS. Use when the user wants a hot news digest, tech trends update, or to check what AI/tech leaders are saying."
license: "MIT"
---

# Hot News Crawler

Aggregates real-time trending content from 6 hot-list platforms + 25+ AI/tech leader RSS feeds. Every run fetches only current data — no stale content.

## Quick Start

```bash
python scripts/crawl.py
```

This runs all crawlers and outputs a Slack-formatted digest. For specific sections only:

```bash
python scripts/crawl.py --sources hackernews,github,leaders
python scripts/crawl.py --sources leaders --category ai
python scripts/crawl.py --sources leaders --category tech
```

## Available Sources

### Hot Lists (real-time trending)

| Source | Flag | Method | Auth |
|--------|------|--------|------|
| HackerNews | `hackernews` | Firebase API | None |
| Lobste.rs | `lobsters` | Public JSON | None |
| GitHub Trending | `github` | HTML scrape | None |
| Product Hunt | `producthunt` | RSS feed | None |
| TechMeme | `techmeme` | RSS feed | None |
| NewsAPI.org | `newsapi` | REST API | `NEWSAPI_KEY` env var |

### AI/Tech Leaders (blog & video tracking)

| Source | Flag | Content |
|--------|------|---------|
| AI leader blogs | `leaders --category ai` | Karpathy, Willison, Lilian Weng, Chip Huyen, Raschka, Mollick, etc. |
| Tech leader blogs | `leaders --category tech` | PG, a16z, Stratechery, Benedict Evans, DHH, etc. |
| AI newsletters | `leaders --category newsletters` | Import AI, Interconnects, Latent Space, etc. |
| YouTube channels | `leaders --category youtube` | Karpathy, Lex Fridman, Yannic Kilcher, AI Explained, etc. |

## Configuration

- `NEWSAPI_KEY` environment variable required for NewsAPI module
- Edit `scripts/sources.yaml` to add/remove RSS feeds or adjust limits
- Output is Slack-formatted by default (`<url|title>` links). Pass `--format plain` for terminal-friendly output.

## Adding a New Source

1. Add entry to `scripts/sources.yaml` under the appropriate category
2. For custom crawlers (non-RSS), add a module in `scripts/crawlers/` with a `crawl(**kwargs) -> list[dict]` function
3. Register it in `scripts/crawl.py`

## Pitfalls

- GitHub Trending relies on HTML structure (`article.Box-row`) — may break if GitHub redesigns
- NewsAPI free tier: 100 req/day, article content truncated to 200 chars
- X/Twitter: NOT available without paid API ($100/mo minimum). All mirror sites (Nitter, xcancel) are dead as of 2026
- Some RSS feeds may go stale if the author stops posting — `--sources leaders` skips entries older than 30 days by default
- Product Hunt RSS can be intermittently empty