{"eid":"commoncrawl","name":"Common Crawl","domain":"commoncrawl.org","resolved_by":"domain","topics":["crawler-scraper","storage"],"n_records":2,"best_vorder":1,"review":null,"aliases":["Common Crawl","Common Crawl (the petabyte corpus — read side only)","commoncrawl.org"],"records":[{"rid":"crawler-scraper/commoncrawl","card":{"rid":"crawler-scraper/commoncrawl","eid":"commoncrawl","topic":"crawler-scraper","slug":"commoncrawl","provider":"Common Crawl","url":"https://commoncrawl.org/","domain":"commoncrawl.org","category":"archive","tier":null,"verdict":"qualifies-with-caveat","vorder":1,"grp":"usable","self_hosted":0,"confidence":"measured","binding":"request-rate","identity_kind":"anonymous","identity_basis":"stated","needs_card":0,"needs_phone":0,"needs_id":0,"needs_institution":0,"country":null,"crypto":null,"interfaces":"rest","egress_metered":0,"per_object_bytes":null,"retention_days":null,"verified_at":"2026-11-04T05:30:27.069Z","verified_basis":"live","checked_on":"2026-11-04","stale_on":"2027-01-04","n_evidence":26,"usd_month":null,"usd_per_tb_month":null,"summary":"keyless index and WARC access","duplicate_of":null,"n_unknown":2,"free_unit":"requests"},"record":{"topic":"crawler-scraper","slug":"commoncrawl","provider":"Common Crawl","url":"https://commoncrawl.org/","category":"archive","verdict":"qualifies-with-caveat","confidence":"measured","confidence_note":"","verified_at":"2026-11-04T05:30:27.069Z","free_tier":{"stated":"keyless index and WARC access","unit":"requests","metered_on":"request-count","source_url":"https://commoncrawl.org/","read_on":"2026-10-22","notes":""},"per_object_cap":{"stated":"not applicable: the unit is a WARC record","bytes":null,"source_url":"https://commoncrawl.org/","notes":""},"retention":{"stated":"one index per crawl, all still listed: collinfo.json names 129 crawl indexes, from CC-MAIN-2008-2009 to CC-MAIN-2026-43","days":null,"source_url":"https://index.commoncrawl.org/collinfo.json","notes":""},"egress":{"terms":"No charge and no account for HTTPS downloads of the data and index files from data.commoncrawl.org.","metered":false,"free_allowance":null,"price_per_tb":null,"source_url":"https://commoncrawl.org/"},"interface":["rest"],"interface_detail":"Index API per crawl on index.commoncrawl.org; WARC files on data.commoncrawl.org","tooling":"Any HTTP client: a CDXJ lookup on index.commoncrawl.org returns a WARC filename, offset and length, and a byte-range GET on data.commoncrawl.org returns that one record.","chaining":"A CDXJ lookup on index.commoncrawl.org names the WARC file, offset and length of a stored capture and one byte-range GET on data.commoncrawl.org reads it, so a page Common Crawl captured is read from the crawl copy when a live fetch fails; for broad or large-scale URL filtering Common Crawl points callers to its URL Index (formerly the Columnar Index) through Amazon Athena or Apache Spark instead of the CDXJ API","limits":[{"kind":"request-rate","value":"at most 1 request a second (HTTP level) on index.commoncrawl.org: the rate a Common Crawl engineer asked index callers to keep to on the Common Crawl user forum in April 2023","source":"https://groups.google.com/g/common-crawl/c/d7ftvUSuGYI"},{"kind":"request-rate","value":"no concurrent requests from one IP address on the CDXJ API: callers are asked to sleep between calls, not to run multiple threads at once on the same IP and not to use proxy networks; an HTTP 503 answer means the acceptable request rate is exceeded","source":"https://commoncrawl.org/faq"},{"kind":"request-rate","value":"the CDXJ API on index.commoncrawl.org is heavily rate limited; an IP address that sends too many requests may be blocked, and a blocked IP address should wait 24 hours","source":"https://commoncrawl.org/faq"}],"binding":"request-rate","limits_note":"","identity_cost":{"kind":"anonymous","detail":"keyless","country":null},"reliability_note":"The CDXJ index is intermittent: on 2026-10-30 the same lookup answered 504 Gateway Time-out from nginx after 11.0 s and 200 after 9.1 s, while a WARC byte-range read on data.commoncrawl.org answered 206 in 0.7 s.","price_per_tb_month":null,"evidence":[{"url":"https://index.commoncrawl.org/CC-MAIN-2026-38-index?url=example.com&output=json&limit=1","date":"2026-10-22","proved":"504 Gateway Time-out after 11.3 s from nginx: the index did not answer"},{"url":null,"date":"2026-10-22","proved":"state failing, 9 consecutive probe timeouts, the only provider not ok"},{"url":"probe:https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1","date":"2026-10-30","proved":"the CDXJ lookup answered 200 after 9.1 s with no key","capture":"sha256:cdf99b6983e5ed4af67d1e02e05b9617152498aee0d3d05458a01fc04dac1139","quote":"\"elapsed_ms\": 9101.5"},{"url":"probe:https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1","date":"2026-10-30","proved":"the same CDXJ lookup answered 504 Gateway Time-out from nginx after 11.0 s, minutes before the 200","capture":"sha256:e52ddd1e86b2a670851fd33b6eb0841152bf9c9c189b259fb88e95f6b6411420","quote":"\"status\": 504"},{"url":"probe:https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz","date":"2026-10-30","proved":"an anonymous byte-range GET on data.commoncrawl.org returned the one WARC record the index named (206)","capture":"sha256:72f6da208a9a3a417c5c03d2933225a8bb0fcc05ca90649838c39eb3e9520b1f","quote":"\"content_range\": \"bytes 937296400-937320567/958726164\""},{"url":"probe:https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=example.com&output=json&limit=1","date":"2026-10-30","proved":"a second CDXJ lookup on the October 2026 index answered 504 Gateway Time-out","capture":"sha256:eb96ce4ec2b1b5024c01245586a0465200b5826a2172b268544712b16944488c","quote":"\"status\": 504"},{"url":"probe:https://index.commoncrawl.org/collinfo.json","date":"2026-10-30","proved":"collinfo.json lists one index per crawl, back to the 2008-2009 ARC crawl; the body has 129 entries","capture":"sha256:a683312f4448cdda63b38f40c7289853fa82eb1e7e3cd5a3674e46142b01891a","quote":"\"id\": \"CC-MAIN-2008-2009\""},{"url":"probe:https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1","date":"2026-10-30","proved":"the index answer names the WARC filename, offset and length of the capture","capture":"sha256:a7e2ae3a76bf30b4972c716e32b6f3c5bdc685e560c44334905c3f37e1c9cdd8","quote":"\"offset\": \"937296400\""},{"url":"https://index.commoncrawl.org/","date":"2026-10-30","proved":null,"capture":"sha256:7ff7835462eec34c002d6da07b91f1a3798465091a93f407303955d644ce79bf","quote":"All data and index files are free to download."},{"url":"https://commoncrawl.org/get-started","date":"2026-10-30","proved":"HTTPS downloads from data.commoncrawl.org need no account","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"There is no need to create an AWS account in order to access the data using this method."},{"url":"https://commoncrawl.org/faq","date":"2026-10-30","proved":"a blocked IP address should wait 24 hours","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"If your IP is temporarily blocked, please wait 24 hours before trying again."},{"url":"https://commoncrawl.org/faq","date":"2026-10-30","proved":"the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"Our CDX API endpoint is frequently abused and therefore heavily rate limited. If your client sends too many requests in a short period of time, your IP address may be temporarily blocked."},{"url":"https://commoncrawl.org/blog/you-can-now-build-directly-on-common-crawl-from-the-browser","date":"2026-10-30","proved":"the index rate limits are strict","capture":"sha256:416b7747fbea2e6bec5566970056be621e9bf9cfdebe8d73d6f69dea95990475","quote":"Please be aware, `index.commoncrawl.org` is rate-limited, and those limits are strict."},{"url":"https://groups.google.com/g/common-crawl/c/d7ftvUSuGYI","date":"2026-10-30","proved":"a Common Crawl engineer asked index users to stay at or below 1 request a second (forum, 2023)","capture":"sha256:00f7c09509cc998d20cbd7370992a850dcecd192518dd55a6695b28cab663616","quote":"reduce the request rate to max. 1 request per second (on HTTP level)."},{"url":"https://commoncrawl.org/blog/oct-nov-2023-performance-issues","date":"2026-10-30","proved":"the 2023 performance post asks small-request callers to stay below 10 requests a second","capture":"sha256:536be8f1769d1477a7b3618e294584068b09b804852b884020115c8a058f4add","quote":"If you’re sending small requests for index information or single webpages contained in WARC files, we can handle a few thousand requests per second total for everyone combined, so you’ll want to stay below 10 per second"},{"url":"probe:https://index.commoncrawl.org/collinfo.json","date":"2026-10-31","proved":"collinfo.json lists one CDXJ index per crawl, back to CC-MAIN-2008-2009; the body has 129 entries, the newest CC-MAIN-2026-43","capture":"sha256:a683312f4448cdda63b38f40c7289853fa82eb1e7e3cd5a3674e46142b01891a","quote":"\"id\": \"CC-MAIN-2008-2009\""},{"url":"probe:https://groups.google.com/g/common-crawl/c/d7ftvUSuGYI","date":"2026-10-31","proved":"a Common Crawl engineer asked index callers on the Common Crawl user forum (April 2023) to keep to at most 1 request a second","capture":"sha256:ad05e0640adfef35a6bf073ca54fd37649f7e5735f44abe1dd00a2933111e2f3","quote":"reduce the request rate to max. 1 request per second (on HTTP level)."},{"url":"probe:https://commoncrawl.org/team","date":"2026-10-31","proved":"Sebastian Nagel, the author of the April 2023 forum post, is on the Common Crawl team as Distinguished Engineer","capture":"sha256:2fe717a7b57d1ff846b91d34c9ab55c87a52e9351b4fb0a6e6e40267f4f0f70d","quote":"Sebastian Nagel Distinguished Engineer"},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"the CDXJ API is to be called with a pause between calls, no parallel threads from one IP address and no proxy networks","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"Please sleep between calls to our API (including if you run your script repeatedly in a loop), don't run multiple threads at once on the same IP, and don't use proxy networks."},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"an HTTP 503 from the CDXJ API means the caller exceeded the acceptable request rate","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"If you receive HTTP 503 responses, please slow down your request rate. These are a sign you've exceeded the acceptable request rate."},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"Common Crawl points broad or large-scale URL filtering to its URL Index, through Amazon Athena or Apache Spark, instead of the CDXJ API","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"You may wish to use our URL Index (previously called the \"Columnar Index\") via Amazon Athena or Apache Spark if your query involves broad or large-scale filtering."},{"url":"probe:https://index.commoncrawl.org/","date":"2026-10-31","proved":"the index server page sends bulk filtering to the Columnar Index","capture":"sha256:7beb65c8ff3f5ed6b0744426593bfae7f2d6640bb8927ef79ac0a07839fe4c8c","quote":"The Columnar Index is a better fit for bulk filtering and aggregation."},{"url":"probe:https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz","date":"2026-10-31","proved":"an anonymous byte-range GET on data.commoncrawl.org answered 206; the October 2026 WARC file is 958726164 bytes","capture":"sha256:0daf334f2fba907a2ee686736b3c76857b68fd5fa6af726c41999143076744d4","quote":"\"content_range\": \"bytes 0-1023/958726164\""},{"url":"probe:https://commoncrawl.org/get-started","date":"2026-10-31","proved":"HTTPS downloads from data.commoncrawl.org need no account","capture":"sha256:25e9bc50af15ca0235cbc0fad6a7fe9463e3bcf37b9233e78236f9ca6578ba7a","quote":"There is no need to create an AWS account in order to access the data using this method."},{"url":"probe:https://index.commoncrawl.org/CC-MAIN-2026-43-index?url=wikipedia.org&output=json&limit=1","date":"2026-10-31","proved":"a keyless CDXJ lookup answered 200 on 2026-10-31 with the WARC filename, offset and length of the capture","capture":"sha256:a7e2ae3a76bf30b4972c716e32b6f3c5bdc685e560c44334905c3f37e1c9cdd8","quote":"\"length\": \"24168\", \"offset\": \"937296400\""},{"url":"https://commoncrawl.org/get-started","date":"2026-11-04","proved":"HTTPS downloads from data.commoncrawl.org need no account, so the free read path is anonymous","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"There is no need to create an AWS account in order to access the data using this method."}],"notes":"Degraded, not dead: on 2026-10-30 the CDXJ index answered 504 and 200 to the same lookup within minutes, while the WARC files on data.commoncrawl.org answered at once. One offer with storage/common-crawl, which reads the same corpus as storage."},"claims":[{"path":"$.free_tier","kind":"has-free-tier","num":1.0,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.free_tier.stated","kind":"free-quota","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"},{"path":"$.limits[0].value","kind":"limit:request-rate","num":1.0,"unit":"request","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.limits[1].value","kind":"limit:request-rate","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"},{"path":"$.limits[2].value","kind":"limit:request-rate","num":24.0,"unit":"hours","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"}],"unknown":[["$.evidence[1].url","private-url"],["$.evidence[8].proved","withheld"]],"history":[],"drift":[]},{"rid":"storage/common-crawl","card":{"rid":"storage/common-crawl","eid":"commoncrawl","topic":"storage","slug":"common-crawl","provider":"Common Crawl (the petabyte corpus — read side only)","url":"https://commoncrawl.org/get-started","domain":"commoncrawl.org","category":"academic","tier":null,"verdict":"fails-quota","vorder":6,"grp":"rejected","self_hosted":0,"confidence":"measured","binding":"request-rate","identity_kind":"anonymous","identity_basis":"defaulted","needs_card":null,"needs_phone":null,"needs_id":null,"needs_institution":null,"country":null,"crypto":null,"interfaces":"rest,s3,cli","egress_metered":0,"per_object_bytes":null,"retention_days":null,"verified_at":"2026-11-04T05:30:27.067Z","verified_basis":"inherited","checked_on":"2026-10-21","stale_on":"2026-12-06","n_evidence":38,"usd_month":null,"usd_per_tb_month":null,"summary":"Read-only corpus, free to download over HTTPS with no account; the October 2026 crawl alone is 2.16 billion pages, 109.20 TiB compressed. It holds Common Cra...","duplicate_of":"crawler-scraper/commoncrawl","n_unknown":4,"free_unit":"mixed"},"record":{"topic":"storage","slug":"common-crawl","provider":"Common Crawl (the petabyte corpus — read side only)","url":"https://commoncrawl.org/get-started","category":"academic","free_tier":{"stated":"Read-only corpus, free to download over HTTPS with no account; the October 2026 crawl alone is 2.16 billion pages, 109.20 TiB compressed. It holds Common Crawl's own crawls and takes no user data.","unit":"mixed","metered_on":"unstated","source_url":"https://data.commoncrawl.org/crawl-data/index.html","read_on":"2026-10-30","notes":""},"per_object_cap":{"stated":"not applicable: the bucket stores no user objects; a read is a whole Common Crawl WARC file (958726164 bytes for one October 2026 file) or a byte range of one","source_url":"https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz","notes":""},"retention":{"stated":"one index per crawl, all still listed: collinfo.json names 129 crawl indexes, from CC-MAIN-2008-2009 to CC-MAIN-2026-43","source_url":"https://index.commoncrawl.org/collinfo.json","notes":""},"egress":{"terms":"Free and anonymous over HTTPS: 'The data is accessible via the https://data.commoncrawl.org/[...] URL scheme. There is no need to create an AWS account in order to access the data using this method.' S3 access (s3://commoncrawl) requires an authenticated AWS account. New: selected crawls are mirrored into a Hugging Face Storage Bucket with CDN pre-warming.","metered":false,"source_url":"https://commoncrawl.org/get-started"},"interface":["rest","s3","cli"],"interface_detail":"HTTPS GET with byte ranges on data.commoncrawl.org, no account; s3://commoncrawl through the S3 API; a Hugging Face Storage Bucket mirror of selected crawls with the same directory layout","tooling":"cURL or wget against https://data.commoncrawl.org/, the AWS CLI against s3://commoncrawl/, the hf CLI against the Hugging Face Storage Bucket mirror, and Common Crawl's official downloader client cc-downloader","chaining":"A read source, not a store: the crawl data is downloaded over HTTPS from data.commoncrawl.org, through the S3 API or from the Hugging Face Storage Bucket mirror into storage or compute held elsewhere, and its CDXJ index names the byte range of a single capture","limits":[{"kind":"request-rate","value":"the CDXJ API on index.commoncrawl.org is heavily rate limited; an IP address that sends too many requests may be blocked, and a blocked IP address should wait 24 hours","source":"https://commoncrawl.org/faq"},{"kind":"request-rate","value":"under load the bucket answers 503 Slow Down to every caller; whole-file downloads recover when retried no more than once per second, index lookups and byte-range reads of single captures do not","source":"https://status.commoncrawl.org/"}],"limits_note":"An unsigned GET to the S3 endpoint commoncrawl.s3.amazonaws.com answered 403 on 2026-10-30 and again on 2026-11-03, while HTTPS on data.commoncrawl.org and the Hugging Face bucket mirror served the same WARC file list with no account. The Get Started page still shows --no-sign-request commands in a second AWS CLI section.","identity_cost":{"kind":"anonymous","detail":"no account for HTTPS downloads from data.commoncrawl.org","country":null},"reliability_note":"Under load the bucket answers 503 Slow Down to every caller, even one sending a single request; whole-file downloads recover with retries no more than once per second, index lookups and byte-range reads of single captures do not.","confidence":"measured","confidence_note":"read path probed: a CDXJ lookup, WARC byte-range reads on data.commoncrawl.org, an unsigned S3 GET and the Hugging Face mirror; the size of one WARC file measured by a range read; the FAQ and get-started bodies read by probe for the verdict, tooling and chaining","evidence":[{"url":"https://commoncrawl.org/get-started","date":"2026-10-21","proved":"'There is no need to create an AWS account in order to access the data using this method'; 'access to data from the Amazon cloud using the S3 API is only allowed for authenticated users'"},{"url":"https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-17/warc.paths.gz","date":"2026-10-21","proved":"probe: GET -> 200, 201,605 bytes, gzip (anonymous listing manifest)"},{"url":"https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-17/warc.paths.gz","date":"2026-10-21","proved":"probe: GET -> 403 (S3 path is auth-only)"},{"url":"probe:https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz","date":"2026-10-30","proved":"an anonymous byte-range GET on data.commoncrawl.org returned the one WARC record the index named (206)","capture":"sha256:72f6da208a9a3a417c5c03d2933225a8bb0fcc05ca90649838c39eb3e9520b1f","quote":"\"content_range\": \"bytes 937296400-937320567/958726164\""},{"url":"https://commoncrawl.org/get-started","date":"2026-10-30","proved":"HTTPS downloads from data.commoncrawl.org need no account","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"There is no need to create an AWS account in order to access the data using this method."},{"url":"probe:https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/warc.paths.gz","date":"2026-10-30","proved":"the October 2026 WARC file list downloads anonymously over HTTPS","capture":"sha256:c52a5232ad8b38a72e9ff5f9229acc4c4fb6a6c2880d9fe5925388be3191c6b5","quote":"\"status\": 200"},{"url":"https://data.commoncrawl.org/crawl-data/index.html","date":"2026-10-30","proved":"the October 2026 crawl is 2.16 billion pages and 109.20 TiB compressed","capture":"sha256:ba5edfd1cbe0dddf181010bed1c6bb91e5b956a79ebb7e8b33a265f278cb5942","quote":"CC-MAIN-2026-43 | October 2026 | 2.16 | 109.20"},{"url":"https://commoncrawl.org/blog/october-2026-crawl-archive-now-available","date":"2026-10-30","proved":"the October 2026 crawl holds 2.16 billion pages, 371.19 TiB uncompressed","capture":"sha256:01b285b5aac86192645576ca8705c06bb1b46c0b6e937135ee1b711cfb0d021f","quote":"The data was crawled between October 5th and October 18th, and contains 2.16 billion web pages (or 371.19 TiB of uncompressed content)."},{"url":"https://commoncrawl.org/faq","date":"2026-10-30","proved":"a blocked IP address should wait 24 hours","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"If your IP is temporarily blocked, please wait 24 hours before trying again."},{"url":"https://status.commoncrawl.org/","date":"2026-10-30","proved":"whole-file downloads recover from 503 by retrying no more than once per second","capture":"sha256:2e34822f6633068569cc7442aa799d06a03d06dc821806bd623b32a9c0dcddea","quote":"For bulk downloads, such as whole files, it’s possible to work around these 503s by politely retrying many times. Retrying no more than once per second is polite enough."},{"url":"https://status.commoncrawl.org/","date":"2026-10-30","proved":"index lookups and byte-range reads of single captures do not recover by retrying","capture":"sha256:2e34822f6633068569cc7442aa799d06a03d06dc821806bd623b32a9c0dcddea","quote":"This retry technique does not work well enough for partial file downloads, such as index lookups and downloading individual webpage captures from within a WARC file."},{"url":"https://commoncrawl.org/faq","date":"2026-10-30","proved":"the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"Our CDX API endpoint is frequently abused and therefore heavily rate limited. If your client sends too many requests in a short period of time, your IP address may be temporarily blocked."},{"url":"https://status.commoncrawl.org/","date":"2026-10-30","proved":"a whole-file download that gets through runs without further rate checks","capture":"sha256:2e34822f6633068569cc7442aa799d06a03d06dc821806bd623b32a9c0dcddea","quote":"And once you are lucky and get a request through, you’ll get the entire file without any further rate limit checks."},{"url":"probe:https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-43/warc.paths.gz","date":"2026-10-30","proved":"an unsigned GET on the S3 endpoint of the commoncrawl bucket is refused","capture":"sha256:bd6aaef88197dbdc681ebc2997ee56b85147f25b105580a5180c8edcc0d25982","quote":"\"status\": 403"},{"url":"probe:https://huggingface.co/buckets/commoncrawl/commoncrawl/resolve/crawl-data/CC-MAIN-2026-43/warc.paths.gz","date":"2026-10-30","proved":"the Hugging Face bucket mirror serves the same 201296-byte WARC file list with no token","capture":"sha256:29abf7676f9ac94af387ba136ebbe196f160e8a355ed5cd992e0d0bf8bfaf30b","quote":"\"status\": 200"},{"url":"https://commoncrawl.org/get-started","date":"2026-10-30","proved":"the S3 API path is stated as for authenticated users (the same page also documents --no-sign-request)","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"Please note, access to data from the Amazon cloud using the S3 API is only allowed for authenticated users."},{"url":"https://status.commoncrawl.org/","date":"2026-10-30","proved":"under load the bucket answers 503 to every caller","capture":"sha256:2e34822f6633068569cc7442aa799d06a03d06dc821806bd623b32a9c0dcddea","quote":"Once the bucket is temporarily overwhelmed, it sends these 503 errors to everyone, including users sending even a single request."},{"url":"https://commoncrawl.org/blog/april-2026-crawl-archive-now-available-in-a-hugging-face-storage-bucket","date":"2026-10-30","proved":"the bucket is Common Crawl's own distribution of its corpus, hosted under the AWS Open Data Sponsorship Program","capture":"sha256:6e6e564218575ec7e202c55511eb0a2cd9c3774f8438475cd931531293e08f8b","quote":"AWS S3 remains the canonical distribution point for the Common Crawl corpus, generously hosted through Amazon Web Services' Open Data Sponsorship Program"},{"url":"https://commoncrawl.org/get-started","date":"2026-10-30","proved":"a Hugging Face Storage Bucket mirrors selected crawls with the S3 layout","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"Selected crawl archives are mirrored in a Hugging Face Storage Bucket, using the same directory layout as S3"},{"url":"probe:https://index.commoncrawl.org/collinfo.json","date":"2026-10-30","proved":"collinfo.json lists one index per crawl, back to the 2008-2009 ARC crawl; the body has 129 entries","capture":"sha256:a683312f4448cdda63b38f40c7289853fa82eb1e7e3cd5a3674e46142b01891a","quote":"\"id\": \"CC-MAIN-2008-2009\""},{"url":"probe:https://commoncrawl.org/get-started","date":"2026-10-31","proved":"the crawl data is processed in the AWS cloud or downloaded over HTTPS at no charge","capture":"sha256:25e9bc50af15ca0235cbc0fad6a7fe9463e3bcf37b9233e78236f9ca6578ba7a","quote":"You may process the data in the AWS cloud or download it for free over HTTP(S) with a good Internet connection."},{"url":"probe:https://index.commoncrawl.org/collinfo.json","date":"2026-10-31","proved":"collinfo.json lists one CDXJ index per crawl, back to CC-MAIN-2008-2009; the body has 129 entries, the newest CC-MAIN-2026-43","capture":"sha256:a683312f4448cdda63b38f40c7289853fa82eb1e7e3cd5a3674e46142b01891a","quote":"\"id\": \"CC-MAIN-2008-2009\""},{"url":"probe:https://data.commoncrawl.org/crawl-data/CC-MAIN-2026-43/segments/1791170053823.21/warc/CC-MAIN-20261005105623-20261005135623-00993.warc.gz","date":"2026-10-31","proved":"an anonymous byte-range GET on data.commoncrawl.org answered 206; the October 2026 WARC file is 958726164 bytes","capture":"sha256:0daf334f2fba907a2ee686736b3c76857b68fd5fa6af726c41999143076744d4","quote":"\"content_range\": \"bytes 0-1023/958726164\""},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"cc-downloader is Common Crawl's official downloader client","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"We also provide an official downloader client cc-downloader which is robust and polite."},{"url":"probe:https://commoncrawl.org/get-started","date":"2026-10-31","proved":"any HTTP download agent (the page names cURL and wget) reads the data from outside AWS","capture":"sha256:25e9bc50af15ca0235cbc0fad6a7fe9463e3bcf37b9233e78236f9ca6578ba7a","quote":"you can use any HTTP download agent, such as"},{"url":"probe:https://commoncrawl.org/get-started","date":"2026-10-31","proved":"the hf CLI copies files from the Hugging Face Storage Bucket mirror","capture":"sha256:25e9bc50af15ca0235cbc0fad6a7fe9463e3bcf37b9233e78236f9ca6578ba7a","quote":"hf buckets cp hf://buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/segments/.../warc/CC-MAIN-....warc.gz ./local-path/"},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"the dataset is Common Crawl's own crawl sample of the web; the FAQ's answer to adding a website is the crawler, not an upload","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"Common Crawl's dataset is a sample of the web, and we do not generally archive any entire website but a randomly selected subset of it."},{"url":"probe:https://commoncrawl.org/faq","date":"2026-10-31","proved":"Common Crawl distributes its own copy of the web to readers; it is not a store for their data","capture":"sha256:b8d8611668566f66cbf218403910994c71382a9af8d8278648ed11349dd6ffd1","quote":"Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis."},{"url":"probe:https://commoncrawl.org/get-started","date":"2026-10-31","proved":"HTTPS downloads from data.commoncrawl.org need no account","capture":"sha256:25e9bc50af15ca0235cbc0fad6a7fe9463e3bcf37b9233e78236f9ca6578ba7a","quote":"There is no need to create an AWS account in order to access the data using this method."},{"url":"probe:https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2026-43/warc.paths.gz","date":"2026-11-03","proved":"an unsigned GET on the S3 endpoint of the commoncrawl bucket was refused with 403","capture":"sha256:419053cac96b00fc6360c5a748fefd78d16d00a4517968a4c2220a5f16ecc8c6","quote":"\"status\": 403"},{"url":"https://commoncrawl.org/blog/introducing-cloudfront-access-to-common-crawl-data","date":"2026-11-03","proved":"the S3 API from the Amazon cloud is restricted to authenticated AWS users and unsigned access to s3://commoncrawl/ is disabled","capture":"sha256:f881cf8594afd9f12340ff8c0cfc5372c123906a02a259ee6b51f565f2b23ed4","quote":"will be restricted to authenticated AWS users, and unsigned access to s3://commoncrawl/ will be disabled."},{"url":"https://commoncrawl.org/blog/introducing-cloudfront-access-to-common-crawl-data","date":"2026-11-03","proved":null,"capture":"sha256:f881cf8594afd9f12340ff8c0cfc5372c123906a02a259ee6b51f565f2b23ed4","quote":null},{"url":"https://groups.google.com/g/common-crawl/c/KL64eJzq304","date":"2026-11-03","proved":"a Common Crawl reply on the user forum states the S3 API needs an AWS account, from us-east-1","capture":"sha256:f744c3bef07ebc19cf8b79dc11df8e74d270c44e1f67508a93169fa81abd054d","quote":"accessing data on the bucket s3://commoncrawl/ using the S3 API requires an AWS account and should be done only from the AWS region \"us-east-1\"."},{"url":"https://commoncrawl.org/get-started","date":"2026-11-03","proved":"a second AWS CLI section of the Get Started page still shows --no-sign-request commands","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"aws --no-sign-request s3 ls s3://commoncrawl/crawl-data/CC-MAIN-2018-17/segments/1524125937193.1/warc/"},{"url":"https://commoncrawl.org/get-started","date":"2026-11-03","proved":"HTTPS downloads from data.commoncrawl.org need no account","capture":"sha256:f035bf60f3892932f25c16f775fb450fc05385e94f3405fd0ba1b9e5f7714d37","quote":"There is no need to create an AWS account in order to access the data using this method."},{"url":"https://commoncrawl.org/faq","date":"2026-11-03","proved":"the CDXJ index is heavily rate limited and blocks an IP address that sends too many requests, so request-rate is the constraint that binds reads","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"Our CDX API endpoint is frequently abused and therefore heavily rate limited. If your client sends too many requests in a short period of time, your IP address may be temporarily blocked."},{"url":"https://status.commoncrawl.org/","date":"2026-11-03","proved":"index lookups and single-capture reads stay rate bound under load, so request-rate binds reads","capture":"sha256:2e34822f6633068569cc7442aa799d06a03d06dc821806bd623b32a9c0dcddea","quote":"This retry technique does not work well enough for partial file downloads, such as index lookups and downloading individual webpage captures from within a WARC file."},{"url":"https://commoncrawl.org/faq","date":"2026-11-04","proved":"the CDXJ index is rate limited and blocks an IP address that sends too many requests, so request-rate binds","capture":"sha256:eb8cba4c974736f6f2bb2d8eedacf491f2fb960db386bb581669780186d3a1b5","quote":"Our CDX API endpoint is frequently abused and therefore heavily rate limited. If your client sends too many requests in a short period of time, your IP address may be temporarily blocked."}],"verdict":"fails-quota","verified_at":"2026-11-04T05:30:27.067Z","notes":"Fails the storage quota filter: the bucket holds only Common Crawl's own crawls of the web and takes no user data, so it offers no allowance a user can write to. One offer with crawler-scraper/commoncrawl, the same corpus judged as a crawl archive, which keeps qualifies-with-caveat as a read source.","extras":{"dead_end":{"failure_class":"quota","why":"Not a hosting service: ~300 billion web pages across ~100+ monthly crawls, multi-petabyte, free to READ and free to re-host yourself. There is NO path to host your own data with Common Crawl — confirmed, it publishes its own crawls only.","retry_when":null,"first_recorded":"2026-10-21"}},"binding":"request-rate","binding_note":"the CDXJ index on index.commoncrawl.org is heavily rate limited and can block a caller's IP address, which bounds lookups of single captures; a whole-file download, once it starts, runs without further rate checks"},"claims":[{"path":"$.free_tier","kind":"has-free-tier","num":1.0,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.free_tier.stated","kind":"free-quota","num":109.2,"unit":"TiB","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.limits[0].value","kind":"limit:request-rate","num":24.0,"unit":"hours","per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.limits[1].value","kind":"limit:request-rate","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"}],"unknown":[["$.evidence[31].proved","withheld"],["$.evidence[31].quote","withheld"],["$.extras.dead_end.retry_when","process"],["$.identity_cost.kind","identity-defaulted"]],"history":[],"drift":[]}]}