Common Crawl
commoncrawl.org · 2 records in Crawling, Storage · JSON
Records 2
- caveatCommon Crawl
Crawlingfree tieridentity anonymousno cardlimit request-rateevidence measuredstale on 2027-01-04
- fails: quotaCommon Crawl (the petabyte corpus — read side only)
Storagefree tieridentity anonymous?card?limit request-rateevidence measuredstale on 2026-12-06duplicate
Common Crawl
Crawling- Free allowance
- Keyless index and WARC access
- Verdict
- caveatusable, with a named limit or an unprobed read path
- Cost
- free tier
- Identity
- anonymousno card, no phone
- Binding limit
- request-rateat most 1 request a second (HTTP level) on index.commoncrawl.org: the rate a Common Crawl engineer asked index callers to keep to on the Common Crawl user forum in April 2023; no concurrent requests from one IP address on the CDXJ API...
- Evidence
- measured26 sources; a live request was made and its output kept
- Freshness
- stale on 2027-01-04checked 2026-11-04 (live); stale on 2027-01-04
| Free tier | keyless index and WARC access unit requests; metered on request-count |
|---|---|
| Price | none stated |
| Limits |
|
| Identity | anonymous keyless |
| Egress | No charge and no account for HTTPS downloads of the data and index files from data.commoncrawl.org. |
| Per-object cap | not applicable: the unit is a WARC record |
| Retention | one index per crawl, all still listed: collinfo.json names 129 crawl indexes, from CC-MAIN-2008-2009 to CC-MAIN-2026-43 |
| Interface | rest Index API per crawl on index.commoncrawl.org; WARC files on data.commoncrawl.org |
| Tooling | Any HTTP client: a CDXJ lookup on index.commoncrawl.org returns a WARC filename, offset and length, and a byte-range GET on data.commoncrawl.org returns that one record. |
| Composes with | A CDXJ lookup on index.commoncrawl.org names the WARC file, offset and length of a stored capture and one byte-range GET on data.commoncrawl.org reads it, so a page Common Crawl captured is read from the crawl copy when a live fetch fails; for broad or large-scale URL filtering Common Crawl points callers to its URL Index (formerly the Columnar Index) through Amazon Athena or Apache Spark instead of the CDXJ API |
| Reliability | The CDXJ index is intermittent: on 2026-10-30 the same lookup answered 504 Gateway Time-out from nginx after 11.0 s and 200 after 9.1 s, while a WARC byte-range read on data.commoncrawl.org answered 206 in 0.7 s. |
| Notes | Degraded, not dead: on 2026-10-30 the CDXJ index answered 504 and 200 to the same lookup within minutes, while the WARC files on data.commoncrawl.org answered at once. One offer with storage/common-crawl, which reads the same corpus as storage. |
| Parsed numbers |
|
Evidence 26
| date | source | what it proved |
|---|---|---|
| 2026-10-22 | index.commoncrawl.org/CC-MAIN-2026-38-index?url=example.com& | 504 Gateway Time-out after 11.3 s from nginx: the index did not answer |
| 2026-10-22 | source not published | state failing, 9 consecutive probe timeouts, the only provider not ok |
| 2026-10-30 | probe: https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1 | the CDXJ lookup answered 200 after 9.1 s with no key |
| 2026-10-30 | probe: https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1 | the same CDXJ lookup answered 504 Gateway Time-out from nginx after 11.0 s, minutes before the 200 |
| 2026-10-30 | probe: https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz | an anonymous byte-range GET on data.commoncrawl.org returned the one WARC record the index named (206) |
| 2026-10-30 | probe: https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=example.com&output=json&limit=1 | a second CDXJ lookup on the October 2026 index answered 504 Gateway Time-out |
| 2026-10-30 | probe: https://index.commoncrawl.org/collinfo.json | collinfo.json lists one index per crawl, back to the 2008-2009 ARC crawl; the body has 129 entries |
| 2026-10-30 | probe: https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1 | the index answer names the WARC filename, offset and length of the capture |
| 2026-10-30 | index.commoncrawl.org/ | not published |
| 2026-10-30 | commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account |
| 2026-10-30 | commoncrawl.org/faq | part of the text of Limits, above |
| 2026-10-30 | commoncrawl.org/faq | the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests |
| 2026-10-30 | commoncrawl.org/blog/you-can-now-build-directly-on-common-cr | the index rate limits are strict |
| 2026-10-30 | groups.google.com/g/common-crawl/c/d7ftvUSuGYI | a Common Crawl engineer asked index users to stay at or below 1 request a second (forum, 2023) |
| 2026-10-30 | commoncrawl.org/blog/oct-nov-2023-performance-issues | the 2023 performance post asks small-request callers to stay below 10 requests a second |
| 2026-10-31 | probe: https://index.commoncrawl.org/collinfo.json | collinfo.json lists one CDXJ index per crawl, back to CC-MAIN-2008-2009; the body has 129 entries, the newest CC-MAIN-2026-43 |
| 2026-10-31 | probe: https://groups.google.com/g/common-crawl/c/d7ftvUSuGYI | a Common Crawl engineer asked index callers on the Common Crawl user forum (April 2023) to keep to at most 1 request a second |
| 2026-10-31 | probe: https://commoncrawl.org/team | Sebastian Nagel, the author of the April 2023 forum post, is on the Common Crawl team as Distinguished Engineer |
| 2026-10-31 | probe: https://commoncrawl.org/faq | the CDXJ API is to be called with a pause between calls, no parallel threads from one IP address and no proxy networks |
| 2026-10-31 | probe: https://commoncrawl.org/faq | an HTTP 503 from the CDXJ API means the caller exceeded the acceptable request rate |
| 2026-10-31 | probe: https://commoncrawl.org/faq | Common Crawl points broad or large-scale URL filtering to its URL Index, through Amazon Athena or Apache Spark, instead of the CDXJ API |
| 2026-10-31 | probe: https://index.commoncrawl.org/ | the index server page sends bulk filtering to the Columnar Index |
| 2026-10-31 | probe: https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz | an anonymous byte-range GET on data.commoncrawl.org answered 206; the October 2026 WARC file is 958726164 bytes |
| 2026-10-31 | probe: https://commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account |
| 2026-10-31 | probe: https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1 | a keyless CDXJ lookup answered 200 on 2026-10-31 with the WARC filename, offset and length of the capture |
| 2026-11-04 | commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account, so the free read path is anonymous |
- Free allowance
- Read-only corpus, free to download over HTTPS with no account; the October 2026 crawl alone is 2.16 billion pages, 109.20 TiB compressed.
- Verdict
- fails: quotathe free allowance is too small to matter
- Cost
- free tier
- Identity
- anonymous?placeholder, not evidence; card unknown, phone unknown
- Binding limit
- request-ratethe CDXJ API on index.commoncrawl.org is heavily rate limited; an IP address that sends too many requests may be blocked, and a blocked IP address should wait 24 hours; under load the bucket answers 503 Slow Down to every caller...
- Evidence
- measured38 sources; a live request was made and its output kept
- Freshness
- stale on 2026-12-06checked 2026-10-21 (inherited); stale on 2026-12-06
| Free tier | Read-only corpus, free to download over HTTPS with no account; the October 2026 crawl alone is 2.16 billion pages, 109.20 TiB compressed. It holds Common Crawl's own crawls and takes no user data. unit mixed; metered on unstated |
|---|---|
| Price | none stated |
| Limits |
An unsigned GET to the S3 endpoint commoncrawl.s3.amazonaws.com answered 403 on 2026-10-30 and again on 2026-11-03, while HTTPS on data.commoncrawl.org and the Hugging Face bucket mirror served the same WARC file list with no account. The Get Started page still shows --no-sign-request commands in a second AWS CLI section. |
| Binding | request-rate the CDXJ index on index.commoncrawl.org is heavily rate limited and can block a caller's IP address, which bounds lookups of single captures; a whole-file download, once it starts, runs without further rate checks |
| Identity | anonymous (placeholder) no account for HTTPS downloads from data.commoncrawl.org |
| Egress | Free and anonymous over HTTPS: 'The data is accessible via the https://data.commoncrawl.org/[...] URL scheme. There is no need to create an AWS account in order to access the data using this method.' S3 access (s3://commoncrawl) requires an authenticated AWS account. New: selected crawls are mirrored into a Hugging Face Storage Bucket with CDN pre-warming. |
| Per-object cap | not applicable: the bucket stores no user objects; a read is a whole Common Crawl WARC file (958726164 bytes for one October 2026 file) or a byte range of one |
| Retention | one index per crawl, all still listed: collinfo.json names 129 crawl indexes, from CC-MAIN-2008-2009 to CC-MAIN-2026-43 |
| Interface | rest, s3, cli HTTPS GET with byte ranges on data.commoncrawl.org, no account; s3://commoncrawl through the S3 API; a Hugging Face Storage Bucket mirror of selected crawls with the same directory layout |
| Tooling | cURL or wget against https://data.commoncrawl.org/, the AWS CLI against s3://commoncrawl/, the hf CLI against the Hugging Face Storage Bucket mirror, and Common Crawl's official downloader client cc-downloader |
| Composes with | A read source, not a store: the crawl data is downloaded over HTTPS from data.commoncrawl.org, through the S3 API or from the Hugging Face Storage Bucket mirror into storage or compute held elsewhere, and its CDXJ index names the byte range of a single capture |
| Reliability | Under load the bucket answers 503 Slow Down to every caller, even one sending a single request; whole-file downloads recover with retries no more than once per second, index lookups and byte-range reads of single captures do not. |
| Notes | Fails the storage quota filter: the bucket holds only Common Crawl's own crawls of the web and takes no user data, so it offers no allowance a user can write to. One offer with crawler-scraper/commoncrawl, the same corpus judged as a crawl archive, which keeps qualifies-with-caveat as a read source. |
| Evidence note | measured read path probed: a CDXJ lookup, WARC byte-range reads on data.commoncrawl.org, an unsigned S3 GET and the Hugging Face mirror; the size of one WARC file measured by a range read; the FAQ and get-started bodies read by probe for the verdict, tooling and chaining |
| Parsed numbers |
|
Evidence 38
| date | source | what it proved |
|---|---|---|
| 2026-10-21 | commoncrawl.org/get-started | 'There is no need to create an AWS account in order to access the data using this method'; 'access to data from the Amazon cloud using the S3 API is only allowed for authenticated users' |
| 2026-10-21 | data.commoncrawl.org/crawl-data/CC-MAIN-2026-17/warc.paths.g | probe: GET -> 200, 201,605 bytes, gzip (anonymous listing manifest) |
| 2026-10-21 | commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-17/warc | probe: GET -> 403 (S3 path is auth-only) |
| 2026-10-30 | probe: https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz | an anonymous byte-range GET on data.commoncrawl.org returned the one WARC record the index named (206) |
| 2026-10-30 | commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account |
| 2026-10-30 | probe: https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/warc.paths.gz | the October 2026 WARC file list downloads anonymously over HTTPS |
| 2026-10-30 | data.commoncrawl.org/crawl-data/index.html | the October 2026 crawl is 2.16 billion pages and 109.20 TiB compressed |
| 2026-10-30 | commoncrawl.org/blog/october-2026-crawl-archive-now-availabl | the October 2026 crawl holds 2.16 billion pages, 371.19 TiB uncompressed |
| 2026-10-30 | commoncrawl.org/faq | part of the text of Limits, above |
| 2026-10-30 | status.commoncrawl.org/ | whole-file downloads recover from 503 by retrying no more than once per second |
| 2026-10-30 | status.commoncrawl.org/ | index lookups and byte-range reads of single captures do not recover by retrying |
| 2026-10-30 | commoncrawl.org/faq | the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests |
| 2026-10-30 | status.commoncrawl.org/ | a whole-file download that gets through runs without further rate checks |
| 2026-10-30 | probe: https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-43/warc.paths.gz | an unsigned GET on the S3 endpoint of the commoncrawl bucket is refused |
| 2026-10-30 | probe: https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/crawl-data/CC-MAIN-2026-43/warc.paths.gz | the Hugging Face bucket mirror serves the same 201296-byte WARC file list with no token |
| 2026-10-30 | commoncrawl.org/get-started | the S3 API path is stated as for authenticated users (the same page also documents --no-sign-request) |
| 2026-10-30 | status.commoncrawl.org/ | under load the bucket answers 503 to every caller |
| 2026-10-30 | commoncrawl.org/blog/april-2026-crawl-archive-now-available- | the bucket is Common Crawl's own distribution of its corpus, hosted under the AWS Open Data Sponsorship Program |
| 2026-10-30 | commoncrawl.org/get-started | a Hugging Face Storage Bucket mirrors selected crawls with the S3 layout |
| 2026-10-30 | probe: https://index.commoncrawl.org/collinfo.json | collinfo.json lists one index per crawl, back to the 2008-2009 ARC crawl; the body has 129 entries |
| 2026-10-31 | probe: https://commoncrawl.org/get-started | the crawl data is processed in the AWS cloud or downloaded over HTTPS at no charge |
| 2026-10-31 | probe: https://index.commoncrawl.org/collinfo.json | collinfo.json lists one CDXJ index per crawl, back to CC-MAIN-2008-2009; the body has 129 entries, the newest CC-MAIN-2026-43 |
| 2026-10-31 | probe: https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz | an anonymous byte-range GET on data.commoncrawl.org answered 206; the October 2026 WARC file is 958726164 bytes |
| 2026-10-31 | probe: https://commoncrawl.org/faq | cc-downloader is Common Crawl's official downloader client |
| 2026-10-31 | probe: https://commoncrawl.org/get-started | any HTTP download agent (the page names cURL and wget) reads the data from outside AWS |
| 2026-10-31 | probe: https://commoncrawl.org/get-started | the hf CLI copies files from the Hugging Face Storage Bucket mirror |
| 2026-10-31 | probe: https://commoncrawl.org/faq | the dataset is Common Crawl's own crawl sample of the web; the FAQ's answer to adding a website is the crawler, not an upload |
| 2026-10-31 | probe: https://commoncrawl.org/faq | Common Crawl distributes its own copy of the web to readers; it is not a store for their data |
| 2026-10-31 | probe: https://commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account |
| 2026-11-03 | probe: https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-43/warc.paths.gz | an unsigned GET on the S3 endpoint of the commoncrawl bucket was refused with 403 |
| 2026-11-03 | commoncrawl.org/blog/introducing-cloudfront-access-to-common | the S3 API from the Amazon cloud is restricted to authenticated AWS users and unsigned access to s3://commoncrawl/ is disabled |
| 2026-11-03 | commoncrawl.org/blog/introducing-cloudfront-access-to-common | not published |
| 2026-11-03 | groups.google.com/g/common-crawl/c/KL64eJzq304 | a Common Crawl reply on the user forum states the S3 API needs an AWS account, from us-east-1 |
| 2026-11-03 | commoncrawl.org/get-started | a second AWS CLI section of the Get Started page still shows --no-sign-request commands |
| 2026-11-03 | commoncrawl.org/get-started | HTTPS downloads from data.commoncrawl.org need no account |
| 2026-11-03 | commoncrawl.org/faq | the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests, so request-rate is the constraint that binds reads |
| 2026-11-03 | status.commoncrawl.org/ | index lookups and single-capture reads stay rate bound under load, so request-rate binds reads |
| 2026-11-04 | commoncrawl.org/faq | the CDXJ index is rate limited and blocks an IP address that sends too many requests, so request-rate binds |
More fields 1
| dead end | {"failure_class": "quota","why": "Not a hosting service: ~300 billion web pages across ~100+ monthly crawls, multi-petabyte, free to READ and free to re-host yourself. There is NO path to host your own data with Common Crawl — confirmed, it publishes its own crawls only.","retry_when": null,"first_recorded": "2026-10-21"} |
|---|