Site Crawl
Crawl a website from a start URL and package its pages, assets, and optional full-page screenshots into a downloadable ZIP archive.
Between 1 and 200 pages, crawled breadth-first from the start URL.
Billed like the Scrape API: 1 credit per started 10 s over HTTP, 4 credits per 10 s when a browser is needed · About 350 credits reserved
Site Crawl API Code
cURL
site-crawl.sh
# 1. Queue the crawl
curl 'https://api.webscraping.so/api/v1/site-crawl' \
--request POST \
--header 'Content-Type: application/json' \
--header 'X-API-Key: YOUR_API_KEY' \
--data '{
"url": "https://example.com",
"max_pages": 50,
"allow_subdomains": false,
"include_assets": true,
"save_screenshots": false
}'
# 2. Poll until status is succeeded, failed, cancelled, or timed_out
curl 'https://api.webscraping.so/api/v1/site-crawl/CRAWL_ID' --header 'X-API-Key: YOUR_API_KEY'
# 3. Get a fresh signed ZIP download URL
curl 'https://api.webscraping.so/api/v1/site-crawl/CRAWL_ID/download-url' --request POST --header 'X-API-Key: YOUR_API_KEY'No crawl yet
Start a crawl to follow its progress and download the ZIP archive here.