Buckets:

commoncrawl
/

test-bucket

69.1 GB

4,321 files

Updated 9 minutes ago

Ctrl+K

Name	Size	Uploaded	Xet hash
cc-index		9 minutes ago	300 items
crawl-data		9 minutes ago	2,009 items
crawldiagnostics		9 days ago	1,000 items
robotstxt		9 days ago	1,000 items
EOT-2020-with-ranks-v4.parquet	109 MB xet	6 months ago	797e1032
README.md	2.78 kB xet	9 minutes ago	43b9802e
crawl-data-sync-plan.filtered.jsonl	1.51 MB xet	5 months ago	858c527d
crawl-data-sync-plan.filtered2.jsonl	844 kB xet	5 months ago	70a59d05
crawl-data-sync-plan.filtered3.jsonl	557 kB xet	5 months ago	91edaafc
crawl-data-sync-plan.filtered4.jsonl	20.8 kB xet	5 months ago	496b4afa
crawl-data-sync-plan.filtered5.jsonl	20.8 kB xet	5 months ago	496b4afa
crawl-data-sync-plan_001.jsonl	1.68 MB xet	5 months ago	107de4bb
crawl-data-sync-plan_002.jsonl	1.68 MB xet	5 months ago	253c81e8
crawl-data-sync-plan_003.jsonl	1.68 MB xet	5 months ago	dadc4df2
example.txt	12 Bytes xet	7 months ago	47c28a7b
foo.txt	3 Bytes xet	7 months ago	793919ca

README.md

Common Crawl Data

An open repository of web crawl data freely available to researchers, developers, and innovators worldwide.

The archives of the main crawl are released on a monthly basis. For details about the data format and how to access the data, see

The following monthly crawl archives have been released on 🤗 Hugging Face:

ID/Location	Announcement	Billion Pages	Total Size Compressed (TiB)
CC-MAIN-2026-17	April 2026	2.19	106.3
CC-MAIN-2026-21	May 2026	2.16	107.1
CC-MAIN-2026-25	June 2026	2.10	102.7
CC-MAIN-2026-30	July 2026	2.14	105.2
CC-MAIN-2026-34	August 2026	2.14	105.1
CC-MAIN-2026-39	September 2026	2.17	103.4

Common Crawl is a California 501(c)(3) registered non-profit organization.