Home › Directory › Scraping & Browsing › Common Crawl
Common Crawl
Free, open repository of petabytes of web crawl data, useful for research and training datasets.
| Category | Scraping & Browsing |
|---|---|
| Authentication | No auth |
| Pricing | Free |
| Website | https://commoncrawl.org/ |
| Docs | https://commoncrawl.org/get-started |
| Last checked | 2026-10-14 |
| As JSON | /api/v1/directory/common-crawl.json |
Information is summarised in our own words and may be out of date; check the vendor's site for current terms and prices. Not affiliated with Common Crawl. Suggest a correction.
More in Scraping & Browsing
Firecrawl
Turns websites into clean markdown or structured data, with crawl, scrape and map endpoints.
Browserbase
Managed headless browsers in the cloud for agents, with session recording and stealth features.
Playwright
Open-source browser automation library for Chromium, Firefox and WebKit in several languages.
Puppeteer
Node library that drives Chrome or Firefox through DevTools protocols for scraping and testing.