firecrawl

---
name: firecrawl
description: Scrape web pages to clean markdown using Firecrawl v2 — handles JS-heavy pages, site crawls, URL mapping, document parsing (PDF/DOCX/XLSX), LLM-powered extraction, autonomous agent scraping, and post-scrape browser interaction (Interact API). Prefer over WebFetch for quality and completeness. Triggers on scrape URL, fetch page, crawl site, extract content, parse document, web to markdown, DeepWiki, Firecrawl.
license: MIT
---

# Firecrawl & Jina Web Scraping

## Firecrawl vs WebFetch

Prefer `firecrawl scrape URL --only-main-content` over the WebFetch tool—it produces cleaner markdown, handles JavaScript-heavy pages, and avoids content truncation (>80% benchmark coverage). WebFetch is acceptable as a fallback when Firecrawl is unavailable.

```bash
# Preferred approach:
firecrawl scrape https://docs.example.com/api --only-main-content
```

## Token-Efficient Scraping

Inspired by Anthropic's [dynamic filtering](https://claude.com/blog/improved-web-search-with-dynamic-filtering)—always filter before reasoning. This reduced input tokens by ~24% and improved accuracy by ~11% in their benchmarks.

### The Principle: Search → Filter → Scrape → Filter → Reason

**DO:**
```
Search (titles/URLs only) → Evaluate relevance → Scrape top hits → Filter by section → Reason
```
**DON'T:**
```
Search → Scrape everything → Reason over all of it
```

### Step-by-Step Efficient Workflow

```bash
# Step 1: Search — get titles/URLs only (cheap)
firecrawl search "query" --limit 20

# Step 2: Evaluate results, pick 3-5 best URLs

# Step 3: Scrape only those, filter to relevant sections
firecrawl scrape URL1 --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py \
  --sections "API,Authentication" --max-chars 5000
```

### Post-Processing with filter_web_results.py

Pipe any Firecrawl or Exa output through this script to reduce context before reasoning:

```bash
# Extract only matching sections from scraped page
firecrawl scrape URL --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "Pricing,Plans"

# Keep only paragraphs with keywords
firecrawl search "query" --scrape --pretty | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --keywords "pricing,cost" --max-chars 5000

# Extract specific JSON fields from API output
python3 ~/.claude/skills/exa-search/scripts/exa_search.py "query" --json | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --fields "title,url,text" --max-chars 3000

# Combine filters with stats
firecrawl scrape URL --only-main-content | \
  python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py --sections "API" --keywords "endpoint" --compact --stats
```

**Full path:** `python3 ~/.claude/skills/firecrawl/scripts/filter_web_results.py`
**Flags:** `--sections`, `--keywords`, `--max-chars`, `--max-lines`, `--fields` (JSON), `--strip-links`, `--strip-images`, `--compact`, `--stats`

### Other Token-Saving Patterns

- **Use `--only-main-content`** to strip navigation and footer boilerplate, reducing token consumption. Omit only when nav/footer content is specifically needed.
- **Use `--only-clean-content`** (Python API script) for aggressive cleaning—strips nav, ads, and cookie banners. Stronger than `--only-main-content`; use when the page is still noisy after main-content filtering.
- **Use `firecrawl map URL --search "topic"` first** to find relevant subpages before scraping
- **Use `--format links` first** to get URL list, evaluate, then scrape selectively
- **Use `--max-chars`** with `exa_contents.py` to cap extraction length
- **Use `--formats summary`** (Python API script) over full text when you need the gist, not raw content

### Claude API Native Tools (for API Agent Builders)

Anthropic's API now offers built-in dynamic filtering tools:
```
web_search_20260209 / web_fetch_20260209
Header: anthropic-beta: code-execution-web-tools-2026-02-09
```
These have built-in dynamic filtering via code execution. Use them when building Claude API agents directly. Use Firecrawl/Exa when you need: autonomous agents, batch scraping, structured extraction, domain-specific crawling, or when not on the Claude API.

---

## Available Tools

### 1. Official Firecrawl CLI (`firecrawl`) — Primary

**Setup:** `npm install -g firecrawl-cli && firecrawl login --api-key $FIRECRAWL_API_KEY`

| Command | Purpose | Quick Example |
|---------|---------|---------------|
| `scrape` | Single page → markdown | `firecrawl scrape URL --only-main-content` |
| `crawl` | Entire site with progress | `firecrawl crawl URL --wait --progress --limit 50` |
| `map` | Discover all URLs on a site | `firecrawl map URL --search "API"` |
| `search` | Web search (+ optional scrape) | `firecrawl search "query" --limit 10` |

**Full CLI reference:** `references/cli-reference.md`

### 2. Auto-Save Alias (`fc-save`) — Shell Alias

Requires shell alias setup (not bundled with this skill).

```bash
fc-save URL
# → Saves to ~/Desktop/Screencaps & Chats/Web-Scrapes/docs-example-com-api.md
```

### 3. Python API Script (`firecrawl_api.py`) — Advanced Features

**Command:** `python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py <command>`
**Requires:** `FIRECRAWL_API_KEY` env var, `pip install firecrawl-py requests`

| Command | Purpose | Quick Example |
|---------|---------|---------------|
| `search` | Web search with scraping | `firecrawl_api.py search "query" -n 10` |
| `scrape` | Single URL with page actions | `firecrawl_api.py scrape URL --formats markdown summary` |
| `batch-scrape` | Multiple URLs concurrently | `firecrawl_api.py batch-scrape URL1 URL2 URL3` |
| `crawl` | Website crawling | `firecrawl_api.py crawl URL --limit 20` |
| `map` | URL discovery | `firecrawl_api.py map URL --search "query"` |
| `parse` | Parse local documents (PDF, DOCX, XLSX) | `firecrawl_api.py parse report.pdf` |
| `extract` | LLM-powered structured extraction | `firecrawl_api.py extract URL --prompt "Find pricing"` |
| `agent` | Autonomous extraction (no URLs needed) | `firecrawl_api.py agent "Find YC W24 AI startups"` |
| `parallel-agent` | Bulk agent queries (v2.8.0+) | `firecrawl_api.py parallel-agent "Q1" "Q2" "Q3"` |
| `interact` | Post-scrape browser interaction | `firecrawl_api.py interact SCRAPE_ID --prompt "Click pricing"` |
| `interact-stop` | Stop an interact session | `firecrawl_api.py interact-stop SCRAPE_ID` |

**Agent models:** `spark-1-fast` (10 credits, simple), `spark-1-mini` (default), `spark-1-pro` (thorough)

**Full Python API reference:** `references/python-api-reference.md`

### 4. DeepWiki — GitHub Repo Documentation

```bash
~/.claude/skills/firecrawl/scripts/deepwiki.sh <owner/repo> [section] [options]
```

AI-generated wiki for any public GitHub repo. No API key required.

```bash
# Overview
~/.claude/skills/firecrawl/scripts/deepwiki.sh karpathy/nanochat

# Browse sections
~/.claude/skills/firecrawl/scripts/deepwiki.sh langchain-ai/langchain --toc

# Specific section
~/.claude/skills/firecrawl/scripts/deepwiki.sh karpathy/nanochat 4.1-gpt-transformer-implementation

# Full dump for RAG
~/.claude/skills/firecrawl/scripts/deepwiki.sh openai/openai-python --all --save
```

### 5. Jina Reader (`jina`) — Fallback

Use when Firecrawl fails or for **Twitter/X URLs** (Firecrawl blocks Twitter, Jina works).

```bash
jina https://x.com/username/status/123456
```

---

## Firecrawl vs Exa vs Native Claude Tools

| Need | Best Tool | Why |
|------|-----------|-----|
| Single page → markdown | `firecrawl scrape --only-main-content` | Cleanest output |
| Search + scrape in one shot | `firecrawl search --scrape` | Combined operation |
| Crawl entire site | `firecrawl crawl --wait --progress` | Link following + progress |
| Local file → markdown | `firecrawl_api.py parse FILE` | Direct upload, no URL needed |
| Autonomous data finding | `firecrawl_api.py agent` | No URLs needed |
| Semantic/neural search | Exa `exa_search.py` | AI-powered relevance |
| Find research papers | Exa `--category "research paper"` | Academic index |
| Quick research answer | Exa `exa_research.py` | Citations + synthesis |
| Find similar pages | Exa `exa_similar.py` | Competitive analysis |
| Claude API agent building | Native `web_search_20260209` | Built-in dynamic filtering |
| Twitter/X content | `jina URL` | Only tool that works |
| GitHub repo docs | `deepwiki.sh owner/repo` | AI-generated wiki |
| Anti-bot / Cloudflare bypass | `scrapling` stealth fetch | Local Turnstile solver |
| Element-level extraction | `scrapling` + CSS selectors | Precision targeting, adaptive tracking |
| No API key scraping | `scrapling` HTTP fetch | 100% local, no credentials |
| Site redesign resilience | `scrapling` adaptive mode | SQLite similarity matching |
| Budget JS-rendered scrape | `cf_browser.py markdown URL` | CF free tier: 10 min/day, $0.09/hr paid |
| Free static page fetch | `cf_browser.py markdown URL --no-render` | FREE during beta (no JS) |
| Budget multi-page crawl | `cf_browser.py crawl URL` | 5 free crawls/day, 100 pages each |
| Incremental re-crawl | `cf_browser.py crawl --modified-since` | Built-in, Firecrawl lacks this |
| Page screenshot/PDF | `cf_browser.py screenshot/pdf URL` | Built-in CF endpoints, cheaper |
| AI structured extraction | `cf_browser.py json URL --prompt "..."` | Workers AI included free |

---

## Common Workflows

### Single Page Scraping
```bash
firecrawl scrape https://example.com/page --only-main-content
# Or auto-save: fc-save URL
# Or to file: firecrawl scrape URL --only-main-content -o page.md
```

### Documentation Crawling
```bash
# Map first, then crawl relevant paths
firecrawl map https://docs.example.com --search "API"
firecrawl crawl https://docs.example.com --include-paths /api,/guides --wait --progress
```

### Research Workflow
```bash
firecrawl search "machine learning best practices 2026" --scrape --scrape-formats markdown
```

### Document Parsing (Local Files)

Parse local documents into clean Markdown. Use `parse` for local or non-public files; use `scrape` for public URLs pointing to documents—both use the same Rust-based parser.

```bash
# PDF to markdown
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse report.pdf

# Excel spreadsheet with main content only
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse data.xlsx --only-main-content

# Word doc with zero data retention, save to file
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse contract.docx --zero-data-retention -o contract.md

# Raw JSON output for programmatic use
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse invoice.pdf --json
```

Supported formats: PDF, DOCX, DOC, XLSX, XLS, HTML, HTM, ODT, RTF (up to 50 MB).

### PDF Parsing (Fire-PDF v2.9)

Fire-PDF is now the default parsing pipeline for all PDF scrapes. Three modes:

```bash
# Auto mode (default) — detects text layer vs scanned, chooses best method
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://example.com/report.pdf"

# Fast mode — text layer only, skip OCR (use for PDFs with selectable text)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape URL --pdf-mode fast

# OCR mode — force full OCR (use for scanned docs or image-only PDFs)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape URL --pdf-mode ocr

# Limit pages parsed (large documents)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py parse report.pdf --pdf-mode auto --pdf-max-pages 20
```

| Flag | Values | Notes |
|------|--------|-------|
| `--pdf-mode` | `fast`, `auto`, `ocr` | Default: `auto`. `fast` = text layer only; `ocr` = force OCR |
| `--pdf-max-pages` | integer | Caps pages parsed; useful for budget control on large PDFs |

Both flags work on `scrape` (for PDF URLs) and `parse` (for local files).

### Agent-Powered Research (No URLs Needed)
```bash
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py agent \
  "Compare pricing tiers for Firecrawl, Apify, and ScrapingBee"
```

### Interact Workflows (Post-Scrape Browser Interaction)

Scrape a page, then take actions on it—click buttons, fill forms, extract dynamic content. Two modes: AI prompts (natural language) and code execution (Node.js/Python/Bash).

#### When to Use Interact vs. Actions

| Need | Use | Why |
|------|-----|-----|
| Click/wait before a single scrape | `--actions` on scrape | Fire-and-forget, no session overhead |
| Multiple interactions with same page | `interact` | Persistent session, back-and-forth |
| Fill forms, log in, navigate | `interact` | Stateful, multi-step |
| Simple "wait for JS to load" | `--actions` with `wait` | Cheaper, no session |

#### Basic Interact (AI Prompt Mode)
```bash
# Step 1: Scrape and note the Scrape ID from output
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://example.com/pricing"

# Step 2: Interact using natural language
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --prompt "Click the Enterprise pricing tab"

# Step 3: More interactions on same session
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --prompt "What is the monthly price for the Enterprise plan?"

# Step 4: Stop when done
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact-stop SCRAPE_ID
```

#### Code Execution Mode (Cheaper)
```bash
# Execute Playwright code directly (2 credits/min vs 7 for prompts)
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "const text = await page.locator('.pricing-table').textContent(); console.log(text);"

# Python mode
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "text = await page.locator('.content').text_content(); print(text)" \
  --language python
```

#### Persistent Profile (Login Sessions)
```bash
# Scrape with a named profile — browser state persists across sessions
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/login" \
  --profile my-app --json

# Interact to log in
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py interact SCRAPE_ID \
  --code "await page.fill('#email', 'user@example.com'); await page.fill('#password', 'pass'); await page.click('button[type=submit]');"

# Later: scrape another page with same profile — cookies restored
python3 ~/.claude/skills/firecrawl/scripts/firecrawl_api.py scrape "https://app.example.com/dashboard" \
  --profile my-app
```

**Important:** Interact does NOT return page markdown. To get updated content after interaction, use code mode to extract specific elements, or issue a follow-up scrape.

**Full interact reference:** `references/interact-reference.md`

---

## Billing Notes

- **Credit/token unification (v2.9):** Credits and tokens are now unified—15 tokens = 1 credit. All pricing is expressed in credits.
- **Default cache TTL:** Results are cached for 2 days. Use `--max-age 0` (or `maxAge: 0` in API) to force a fresh scrape regardless of cache.
- **`query` format:** Pass `formats=["query"]` (Python API) to get a direct answer (`data.answer`) instead of full markdown. Use for factual lookups where you don't need the full page content.
- **`audio` format:** `formats=["audio"]` returns an MP3 of the page read aloud. Useful for accessibility pipelines or voice interfaces.
- **`wikimedia` engine:** Pass `engine="wikimedia"` in search options to route queries through Wikimedia. Useful for encyclopedic lookups.

## Troubleshooting

```bash
# Check status and credits
firecrawl --status && firecrawl credit-usage

# Re-authenticate
firecrawl logout && firecrawl login --api-key $FIRECRAWL_API_KEY

# Check API key
echo $FIRECRAWL_API_KEY
```

- **Scrape fails:** Try `jina URL`, or add `--wait-for 3000` for JS-heavy sites
- **Async job stuck:** Check with `crawl-status`/`batch-status`, cancel with `crawl-cancel`/`batch-cancel`
- **Disable telemetry:** `export FIRECRAWL_NO_TELEMETRY=1`

---

## Reference Documentation

| File | Contents |
|------|----------|
| `references/cli-reference.md` | Full CLI parameter reference (scrape, crawl, map, search, fc-save, jina, deepwiki) |
| `references/python-api-reference.md` | Full Python API script reference (all commands, SDK examples) |
| `references/firecrawl-api.md` | Firecrawl Search API reference |
| `references/firecrawl-agent-api.md` | Agent API (spark models, parallel agents, webhooks) |
| `references/actions-reference.md` | Page actions for dynamic content (click, write, wait, scroll) |
| `references/interact-reference.md` | Interact API: post-scrape browser interaction (prompt, code, profiles) |
| `references/branding-format.md` | Brand identity extraction (colors, fonts, UI) |

## Test Suite

```bash
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --quick    # Quick validation
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py            # Full suite
python3 ~/.claude/skills/firecrawl/scripts/test_firecrawl.py --test scrape  # Specific test
```