parsew-batch-scrape
Scrape multiple URLs at once asynchronously using the Parsew API. Use when the user wants to scrape a list of URLs, batch scrape multiple pages, scrape several websites, or process many URLs in bulk. Supports 1-100 URLs per batch. For scraping a single URL, use parsew-scrape instead.
Works with
Claude CodeCursorCodex CLIGitHub CopilotGemini CLI
Agent Skills format with YAML frontmatter. Claude Code reads it as-is.
---
name: "parsew-batch-scrape"
description: "Scrape multiple URLs at once asynchronously using the Parsew API. Use when the user wants to scrape a list of URLs, batch scrape multiple pages, scrape several websites, or process many URLs in bulk. Supports 1-100 URLs per batch. For scraping a single URL, use parsew-scrape instead."
license: "MIT"
---
## Authentication
Requires a `PARSEW_API_KEY` environment variable with a secret key (`sr_` prefix).
If the key is not set, tell the user to get one from https://dash.parsew.com and set it:
```bash
export PARSEW_API_KEY=sr_your_secret_key
```
## How It Works
Batch scraping is asynchronous — you submit a batch of URLs, then poll for results.
1. **Submit** — `POST /v1/batch/scrape` with 1–100 URLs. Returns a batch ID immediately.
2. **Poll** — `GET /v1/batch/scrape/{batchId}` until `status` is `completed` or `failed`.
3. **Collect** — Results are in the `data` array of the poll response.
Results expire **24 hours** after the batch completes.
**Cost:** 1 credit per URL, charged upfront when the batch is submitted.
## Usage
### With the SDK (`@parsew/sdk`)
The SDK handles polling automatically via `waitForCompletion()`:
```typescript
import { Parsew } from '@parsew/sdk/server'
const parsew = new Parsew({ apiKey: process.env.PARSEW_API_KEY })
const job = await parsew.batchScrape([
'https://example.com/page1',
'https://example.com/page2',
'https://example.com/page3',
])
console.log(`Batch ${job.id} submitted — ${job.total} URLs, ${job.creditsUsed} credits`)
// Wait for all results (polls every 2s by default)
const result = await job.waitForCompletion()
for (const item of result.data) {
if (item.error) {
console.error(`Failed: ${item.url} — ${item.error}`)
} else {
console.log(`${item.url}: ${item.markdown.slice(0, 100)}...`)
}
}
```
With options:
```typescript
const job = await parsew.batchScrape(urls, {
waitForSelector: '#content',
waitFor: 5000,
pollInterval: 3000, // Poll every 3s (default: 2s)
timeout: 120000, // Give up after 2 minutes (default: no timeout)
})
```
You can also check status manually:
```typescript
const status = await job.getStatus()
console.log(`${status.completed}/${status.total} done, ${status.failed} failed`)
```
### With curl
**Step 1 — Submit the batch:**
```bash
curl -X POST https://api.parsew.com/v1/batch/scrape \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $PARSEW_API_KEY" \
-d '{
"urls": [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3"
]
}'
```
Response (HTTP 202):
```json
{
"id": "batch_clx9abc123",
"url": "https://api.parsew.com/v1/batch/scrape/batch_clx9abc123",
"total": 3,
"creditsUsed": 3
}
```
**Step 2 — Poll for results:**
```bash
curl https://api.parsew.com/v1/batch/scrape/batch_clx9abc123 \
-H "Authorization: Bearer $PARSEW_API_KEY"
```
Response:
```json
{
"id": "batch_clx9abc123",
"status": "completed",
"total": 3,
"completed": 3,
"failed": 0,
"creditsUsed": 3,
"expiresAt": "2025-01-02T12:00:00Z",
"data": [
{
"url": "https://example.com/page1",
"markdown": "# Page 1...",
"html": "<html>...</html>",
"links": ["..."],
"warning": null
}
]
}
```
**Status values:** `scraping` (in progress), `completed` (done), `failed` (all failed).
Poll every 2–5 seconds until status is `completed` or `failed`.
**Pagination:** For large batches, use `offset` and `limit` query parameters (default limit: 100):
```bash
curl "https://api.parsew.com/v1/batch/scrape/batch_clx9abc123?offset=0&limit=50" \
-H "Authorization: Bearer $PARSEW_API_KEY"
```
## Submit Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `urls` | string[] | Yes | 1–100 URLs to scrape |
| `waitForSelector` | string | No | CSS selector to wait for on each page |
| `waitFor` | integer | No | Max wait time per page in ms (1–60,000) |
| `actions` | array | No | Up to 20 browser actions per page |
## Common Pattern: Map Then Batch Scrape
Use `parsew-map` to discover URLs first, then batch scrape the results:
```typescript
const { links } = await parsew.map('https://example.com/blog', {
pattern: '/blog/.*',
limit: 50,
})
const job = await parsew.batchScrape(links)
const result = await job.waitForCompletion()
```
## Error Handling
| Status | Meaning | Action |
|--------|---------|--------|
| 401 | Invalid or missing API key | Check `PARSEW_API_KEY` |
| 429 | Insufficient credits | Check credit balance (1 credit per URL) |
Individual URLs in the batch can fail independently — check each item's `error` field in the results.More General & Other skills
find-skills
vercel-labs/skills
Helps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express interest in extending capabilities. This skill should be used when the user is looking for functionality that might exist as an installable skill.
1.5M
grill-me
mattpocock/skills
A relentless interview to sharpen a plan or design.
972.7k
grill-with-docs
mattpocock/skills
A relentless interview to sharpen a plan or design, which also creates docs (ADR's and glossary) as we go.
828.8k
