crawler

---
name: "crawler"
description: "[Hyper] Investigate websites with Playwriter plus CDP to choose a crawl strategy, capture API/auth evidence, document findings under .hypercore/crawler/[site]/, and generate crawler code only after discovery is grounded."
license: "MIT"
---

# Crawler Skill

> Playwriter exploration -> CDP evidence capture -> Documentation -> Code generation

<output_language>

Default all user-facing deliverables, saved artifacts, reports, plans, generated docs, summaries, handoff notes, commit/message drafts, and validation notes to Korean, even when this canonical skill file is written in English.

Preserve source code identifiers, CLI commands, file paths, schema keys, JSON/YAML field names, API names, package names, proper nouns, and quoted source excerpts in their required or original language.

Use a different language only when the user explicitly requests it, an existing target artifact must stay in another language for consistency, or a machine-readable contract requires exact English tokens. If a localized template or reference exists (for example `*.ko.md` or `*.ko.json`), prefer it for user-facing artifacts.

</output_language>

<purpose>

- Investigate websites before generating crawler code.
- Capture API, auth, selector, network, anti-bot, and method-selection evidence.
- Leave resumable crawl state under `.hypercore/crawler/<ACTION>.json` and site artifacts under `.hypercore/crawler/[site]/`.
- Generate crawler code only after discovery evidence justifies the method.

</purpose>

<routing_rule>

Use `crawler` for reusable crawling flows, site extraction plans, API reverse engineering for crawling, or analysis-backed crawler code.

Do not use it for generic browser automation, one-off page clicking, or document rewriting with no crawl deliverable. For quick one-off extraction, stay lightweight unless the request expands into reusable crawl design.

</routing_rule>

<instruction_contract>

| Field | Contract |
|---|---|
| Intent | Build evidence-backed crawler plans and code after website/API discovery. |
| Trigger | Reusable crawling, scraping, site-wide extraction, crawl strategy, API reverse engineering, auth/network evidence, or crawler code generation. |
| Scope | Own discovery, method selection, evidence artifacts, durable action state, and generated crawler code when justified. |
| Authority | User/project instructions outrank this skill; observed browser/network/CDP evidence and local artifacts are evidence. |
| Evidence | Use Playwriter, Chrome DevTools/CDP, network summaries, selector checks, anti-bot findings, and artifact files. |
| Tools | Use browser/CDP tools and local file writes only as needed; gate credentials, anti-bot bypass, production, and unsafe scraping. |
| Output | Korean findings plus `.hypercore/crawler/` artifacts and crawler code only after evidence supports it. |
| Verification | Validate method choice, artifacts, selectors/API calls, and generated code against captured evidence. |
| Stop condition | Stop when the crawl method is justified, artifacts are written, code is generated only if safe, or blockers are documented. |

</instruction_contract>

<activation_examples>

Positive examples:

- "Scrape product cards from this shop, inspect the API first, then generate a crawler."
- "Figure out how this logged-in dashboard loads data and document the cookies and headers."
- "Analyze this Cloudflare-protected site and recommend the safest crawl approach."

Negative examples:

- "Open this site and click through the signup flow."
- "Rewrite this crawl runbook for readability."

Boundary example:

- "Grab three prices from this public page right now." Prefer lightweight extraction unless the user asks for a reusable crawler or site-wide strategy.

</activation_examples>

---

<trigger_conditions>

| Trigger | Action |
|--------|------|
| Reusable crawling, scraping, or site-wide extraction | Run immediately |
| Site investigation or API reverse engineering for crawling | Start discovery and API interception |
| One-off extraction from a single page | Treat as a boundary case and keep the workflow lightweight unless reusable crawl work is requested |
| Anti-bot bypass or Cloudflare-heavy target | Start with risk checks and Anti-Detect guidance |

</trigger_conditions>

---

<support_file_read_order>

Read support files in this order:

1. Start with [pre-crawl-checklist.md](rules/pre-crawl-checklist.md) before making crawl or code decisions.
2. Use [playwriter-commands.md](rules/playwriter-commands.md) when you need session control, page interaction, visual inspection, or selector validation (Playwright MCP = **driving**).
3. Use [chrome-devtools-mcp.md](rules/chrome-devtools-mcp.md) when you need first-party Chrome DevTools fidelity for live network requests, console errors, performance traces, Lighthouse audits, or memory snapshots (Chrome DevTools MCP = **debugging**).
4. Use [cdp-capture.md](rules/cdp-capture.md) when you need structured network, cookie, token, storage, or rate-limit evidence with lower token cost than full Playwriter snapshots.
5. Use [network-crawling.md](rules/network-crawling.md) when turning Playwriter / chrome-devtools-mcp / CDP evidence into `API.md`, `NETWORK.md`, and raw evidence files.
6. Use [selector-strategies.md](rules/selector-strategies.md) when DOM extraction is still on the table.
7. Use [crawling-patterns.md](rules/crawling-patterns.md) when pagination, authentication, lazy loading, or retries shape the approach.
8. Use [anti-bot-checklist.md](rules/anti-bot-checklist.md) when the target shows blocks, CAPTCHA, Cloudflare, or explicit anti-detect requirements.
9. Use [action-manifest.md](rules/action-manifest.md) when the run needs a durable state file under `.hypercore/crawler/<ACTION>.json`.
10. Use [document-templates.md](rules/document-templates.md) when writing `.hypercore/crawler/[site]/` artifacts.
11. Use [code-templates.md](rules/code-templates.md) only after the method is chosen and the discovery evidence is documented.

</support_file_read_order>

---

<mandatory_reasoning>

## Mandatory Structured Reasoning

- Always perform an internal structured reasoning pass before crawl design, extraction strategy, or code generation decisions.
- Run a structured reasoning pass for each major phase: discovery, method selection, and implementation planning.
- Do not require an external reasoning MCP. If the task is too unclear to reason through locally, stop and report the blocker instead of continuing without a structure plan.

</mandatory_reasoning>

---

<execution_defaults>

- Do discovery before code generation, selector lock-in, or auth assumptions.
- Use Playwriter to reproduce the user-visible flow, then prefer CDP for structured network/auth evidence capture.
- Prefer an API-backed crawler when CDP or fallback browser-network evidence shows a stable endpoint and manageable auth.
- Keep large DOM or accessibility snapshots rare; use them for structure checks and selector validation, not as the default capture surface.
- If CDP attach fails, document the limitation in `ANALYSIS.md` and use Playwriter interception only when the fallback evidence is still sufficient.
- Stop and report blockers when legal constraints, repeated `403/429/503`, CAPTCHA, or strong anti-bot signals make automation unsafe.
- Do not promise `CRAWLER.ts` until the method, auth material, and rate-limit posture are documented.

</execution_defaults>

---

<workflow>

| Phase | Task | Command/Method |
|-------|------|--------|
| **1. Session** | Create session + open page | `playwriter session new` |
| **2. Explore** | Reproduce the page flow with Playwriter | `accessibilitySnapshot`, `screenshotWithAccessibilityLabels` |
| **3. Capture** | Collect network/auth/perf evidence via `chrome-devtools-mcp` (preferred) or CDP fallback | `list_network_requests`, `list_console_messages`, `performance_start_trace`; CDP `Network.*`, `Storage.*`, `Runtime.evaluate` — see [chrome-devtools-mcp.md](rules/chrome-devtools-mcp.md) and [cdp-capture.md](rules/cdp-capture.md) |
| **4. Analyze** | Decide API-first vs DOM-first | [network-crawling.md](rules/network-crawling.md), [selector-strategies.md](rules/selector-strategies.md) |
| **5. Document** | Save findings under `.hypercore/crawler/[site]/` | Write |
| **6. Code** | Generate crawler implementation | [code-templates.md](rules/code-templates.md) |

</workflow>

---

<method_selection>

| Condition | Method | Notes |
|------|------|------|
| API found via CDP or fallback browser-network evidence + simple auth | **`fetch` / `httpx`** | Fastest |
| API + cookie/token required | **`fetch` + Cookie** | Requires expiry handling |
| API + Cloudflare / DataDome / JA3 fingerprinting | **`curl_cffi` (impersonate Chrome)** | Restores TLS/JA3; pair with residential proxy |
| Discovery / live network + perf evidence | **`chrome-devtools-mcp`** | First-party CDP fidelity (network, console, perf trace, Lighthouse) — see [chrome-devtools-mcp.md](rules/chrome-devtools-mcp.md) |
| Page driving / login / lazy-load triggering | **`playwriter`** | "Make the page do the thing" |
| Strong anti-bot (Cloudflare, DataDome) | **Patchright** or **rebrowser-patches** | Patches Chromium / patches `Runtime.Enable` leakage — see [anti-bot-checklist.md](rules/anti-bot-checklist.md) |
| Chromium-specific fingerprinting | **Camoufox** | Firefox-based stealth fork |
| No API (SSR) and no anti-bot | **Playwright DOM** | Parse directly |

</method_selection>

---

<output_structure>

`ACTION.json` preserves intent, current status, capture mode, blockers, output pointers, and the next step. `.hypercore/crawler/[site-name]/` preserves detailed evidence, analysis, and generated code for that site.

Minimum artifact contract:

- `.hypercore/crawler/<ACTION>.json` is required for reusable, blocked, or resumable crawl work.
- `ANALYSIS.md` is always required for reusable crawl work.
- `SELECTORS.md` is required when DOM extraction is used or kept as a fallback path.
- `API.md` is required when API discovery was attempted; document discovered endpoints or the absence of a usable API.
- `NETWORK.md` is required when cookies, tokens, headers, rate limits, or bot-detection signals affect the method.
- `raw/network-summary.json`, `raw/auth-signals.json`, and `raw/endpoint-candidates.json` are recommended when CDP capture is available, and should back the human-readable docs instead of replacing them.
- `CRAWLER.ts` is required only after discovery evidence is written and the chosen method is justified.

Starter interaction commands live in [playwriter-commands.md](rules/playwriter-commands.md). CDP evidence capture lives in [cdp-capture.md](rules/cdp-capture.md). Durable action-state rules live in [action-manifest.md](rules/action-manifest.md). Keep the core focused on method choice, output gates, and stop conditions.

**Templates:** [document-templates.md](rules/document-templates.md)

</output_structure>

---

<blocked_outcomes>

For blocked or unsafe runs:

- write `ANALYSIS.md` with the blocker, the evidence that triggered the stop, and the safest next step
- write `NETWORK.md` when auth signals, block responses, or anti-bot findings affected the decision
- write any available raw evidence files even when the run is blocked, so the stop is auditable
- update `ACTION.json` so `status`, `capture_mode`, blockers, and output pointers match the blocked state
- omit `CRAWLER.ts` until the blocker is resolved or the method becomes safe to automate

</blocked_outcomes>

---

<validation>

```text
✅ Playwriter session created
✅ `ACTION.json` created when the run is reusable, blocked, or resumable
✅ Structure analyzed with limited Playwriter snapshots
✅ CDP capture attempted for network/auth evidence
✅ raw evidence files recorded when CDP capture is available, or the fallback limitation documented when it is not
✅ Selector extraction validated
✅ Findings documented under .hypercore/crawler/
✅ Crawler code generated
✅ Structured reasoning notes recorded for major phases
✅ legal, rate-limit, and bot-detection blockers documented before scaling
✅ blocked runs reported explicitly when crawler code is unsafe or premature
✅ `ACTION.json` status and `site_dir` match the actual run outputs
✅ completed runs leave `ACTION.json.next_step` empty or terminal and point outputs at final files
```

</validation>

---

<forbidden>

| Category | Forbidden |
|------|------|
| **Analysis** | Guess selectors without structure analysis |
| **Approach** | Use DOM-only flow without checking APIs |
| **Documentation** | Skip documenting analysis results |
| **Network** | Ignore rate limiting |

</forbidden>

---

<example>

```bash
# User: /crawler crawl products from https://shop.example.com

# 1. Create durable action context
# .hypercore/crawler/extract-products.json

# 2. Session
playwriter session new  # => 1
playwriter -s 1 -e "state.page = await context.newPage(); await state.page.goto('https://shop.example.com/products')"

# 3. Structure analysis
playwriter -s 1 -e "console.log(await accessibilitySnapshot({ page: state.page }))"
# => list "Products" [ref=e5]: listitem [ref=e6]: link "Product A" [ref=e7]

# 4. CDP capture
playwriter -s 1 -e $'
const client = await state.page.context().newCDPSession(state.page);
await client.send("Network.enable");
state.cdpHits = [];
client.on("Network.responseReceived", (event) => {
  if (event.response.url.includes("/api/")) state.cdpHits.push(event.response.url);
});
'
playwriter -s 1 -e "await state.page.evaluate(() => window.scrollTo(0, 9999))"
playwriter -s 1 -e "console.log(state.cdpHits)"
# => ["/api/products?page=2"]

# 5. Update extract-products.json -> status=running, capture_mode=cdp
# 6. Documentation -> .hypercore/crawler/shop-example-com/ + raw/network-summary.json
# 7. Generate API-based crawler
```

</example>