scraping

---
name: "scraping"
description: "Activate whenever the user wants to: - Scrape data from a website (any topic, any scale) - Extract structured data from HTML pages - Collect URLs across paginated listings - Monitor a site for changes - Build a scraping script in Python - Handle anti-bot protections - Manage proxies for scraping"
license: "MIT"
---

# Scraping Skill

## When to Activate

Activate whenever the user wants to:
- Scrape data from a website (any topic, any scale)
- Extract structured data from HTML pages
- Collect URLs across paginated listings
- Monitor a site for changes
- Build a scraping script in Python
- Handle anti-bot protections
- Manage proxies for scraping

## Workflow (3 Phases)

**Phase 1: Reconnaissance** — always runs
**Phase 2: URL Collection** — optional, ask user
**Phase 3: Detail Scraping** — optional, ask user

---

## Phase 1: Planning

Ask questions **one at a time**. Show the default in `[brackets]`. If the user sends an empty reply or says "default"/"yes"/"ok", silently apply the default and move on.

**Do NOT ask all questions upfront** — follow the branching tree below. Skip questions that are not relevant based on prior answers. API/JS detection happens automatically during recon — never ask the user about those upfront.

### Core Questions (always ask, in this order)

**Q1 — Target URL**
> "What is the target URL?"
→ No default. Required.

**Q2 — Data fields**
> "What data do you want to extract? List the fields you need (e.g. title, price, description, image URL)."
→ No default. Required.

**Q3 — Authentication**
> "Does the site require authentication? [default: No]"
→ If **yes** → ask **Q3a**: "Auth type: username+password / API key / cookie / OAuth token? [default: API key]"
→ If **no** → skip Q3a, continue.

**Q4 — Scope**
> "Scope: single page, paginated listing, or full site crawl? [default: single page]"
→ If **paginated** or **crawl** → ask **Q4a** after Q4: "Deduplicate — skip already-scraped URLs across sessions? [default: Yes]"
→ If **single page** → skip Q4a.

**Q5 — Output format**
> "Output format: JSON / CSV / markdown / raw? [default: JSON]"

**Q6 — Output destination**
> "Save to local file, POST to webhook, or both? [default: local file]"
→ If **webhook** or **both** → ask **Q6a**: "Webhook URL?"

**Q7 — Dry run**
> "Do a dry run first — test on the first page only before the full scrape? [default: Yes]"

### Conditional Questions (only when triggered)

**Q_perf** — ask only after recon confirms Level 3+ site:
> "This site has strong anti-bot protection. Optimize for cost or performance?
> cost / balanced / performance [default: balanced]"
> - **cost**: lightweight models for all agents
> - **balanced**: strong models for complex agents, lightweight for simple ones
> - **performance**: most capable model for every agent
> *(On Claude Code this sets per-agent `model:` — on other platforms it guides global model choice. See `references/model-selection.md`.)*

**Q_proxy** — ask only when Level 4 is recommended:
> "Level 4 requires proxies. Do you have a proxy list or rotating endpoint to provide?"

**Q_sched** — ask at the very end of the session:
> "Do you want to schedule recurring scrapes for this site? [default: No]"
→ If **yes** → ask frequency and time.

---

## Phase 2: Reconnaissance

After planning, invoke the **site-analyzer** agent.

Site-analyzer will:
- Open the target in agent-browser (or Playwright MCP if insufficient)
- Capture all network requests, DOM structure, cookies, session tokens, JS variables
- Detect hidden APIs — convert cURL → Python → test live → confirm with user if found
- Detect WAF/CDN and frontend framework (if relevant to scraping strategy)
- Generate site profile (JSON + markdown) to `.scraping/profiles/<domain>/`
- End with a level recommendation for user confirmation

---

## Decision Tree (6-Factor Priority)

After reconnaissance, evaluate these factors **in order** to pick the starting level:

1. **API available?** → Level 1 with requests + JSON (invoke `api-detection.md`)
2. **Static HTML?** → Level 1 with requests + BeautifulSoup4
3. **Auth required?** → add session/cookie/token layer to chosen approach
4. **Anti-bot detected?** → invoke `anti-bot-advisor` agent; escalate to Level 2 or 3
5. **Data volume?** → add pagination/crawl logic (invoke `pagination-detective`)
6. **Speed vs reliability?** → tune retry count, delay, concurrency

See `references/decision-tree.md` for full flowchart.

---

## Escalation Ladder

| Level | When | Libraries |
|-------|------|-----------|
| 1 | Static HTML or API found | `requests`, `httpx`, `beautifulsoup4` |
| 2 | Level 1 blocked; TLS/header checks | `curl_cffi`, `tls-client`, UA rotation |
| 3 | Level 2 blocked; Cloudflare/DataDome | `scrapling`, `scrapling-enhanced`, `camoufox`, `nodriver`, `botasaurus`, `DrissionPage` |
| 4 | IP blocked; need geo/rotation | proxy rotation layer added to Level 1-3 |

All generated scripts include: retry + exponential backoff, detailed logging, user-configured rate limiting, full error handling.

See `references/escalation-ladder.md` for Python script templates.

---

## Agent Orchestration

| When | Invoke |
|------|--------|
| Start of every session | `site-analyzer` |
| After site-analyzer, before scripting | `data-mapper` |
| Phase 2 (URL collection) | `pagination-detective` |
| Anti-bot response detected | `anti-bot-advisor` |
| CAPTCHA encountered | `captcha-resolver` |
| Level 4 needed | `proxy-manager` |
| After script is generated | `script-reviewer` |
| After data is extracted | `output-formatter` |

---

## CAPTCHA Strategy

Escalate in this order (notify user before each step):
1. Avoid via stealth (Level 2-3 tools)
2. OCR for image/custom CAPTCHAs (pytesseract, easyocr)
3. Library built-ins (scrapling has built-in solver)
4. Paid service (ask user for API key: 2captcha, anti-captcha)
5. Human behavior simulation (Bézier mouse, saved profiles)

Invoke `captcha-resolver` agent when any CAPTCHA is encountered.

---

## Site Profile Management

Site profiles are saved to `.scraping/profiles/<domain>/`:
- `site-profile.json` — machine-readable (reused automatically on next scrape of same domain)
- `site-profile.md` — human-readable for review and editing
- `extraction-config.json` — data-mapper output (selectors, field definitions)

Fingerprint profiles: `.scraping/fingerprints/<domain>.json`
Human behavior profiles: `.scraping/human-profiles/`

---

## Platform Notes

This skill works across Claude Code, Cursor, Codex, OpenCode, and Gemini CLI.
See `references/platform-tools.md` for tool name mappings per platform.