Common Crawl Foundation

non-profit

Verified

https://commoncrawl.org

AI & ML interests

Crawled data and metadata

Recent Activity

malteos updated a bucket 2 days ago

commoncrawl/test-bucket

malteos new activity 4 days ago

commoncrawl/commonlid-results:Add py3langid results on commonlid_nano

malteos new activity 4 days ago

commoncrawl/commonlid-results:Refresh results from results

View all activity

updated a bucket 2 days ago

commoncrawl/test-bucket

in commoncrawl/commonlid-results 4 days ago

Add py3langid results on commonlid_nano

#9 opened 4 days ago by

Refresh results from results

#6 opened 17 days ago by

updated a dataset 5 days ago

commoncrawl/CommonLID

Viewer • Updated 5 days ago • 373k • 176 • 56

updated a bucket 5 days ago

commoncrawl/commoncrawl

in commoncrawl/statistics 8 days ago

Update data (CC-MAIN-2026-39)

#3 opened 8 days ago by

authored 3 papers 10 days ago

Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data

Paper • 2507.00152 • Published Jun 30, 2025 • 1

SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

Paper • 2512.11192 • Published Dec 12, 2025 • 1

CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data

Paper • 2601.18026 • Published Jan 25

updated a bucket 11 days ago

commoncrawl/warc2zip-examples

in commoncrawl/commonlid-results 12 days ago

Add GoogleTranslate-v3 results on commonlid_nano

#8 opened 12 days ago by

Rename GoogleTranslate to GoogleTranslate-v2

#7 opened 12 days ago by

updated 2 datasets 12 days ago

commoncrawl/host-index-testing-v2

Viewer • Updated 12 days ago • 9B • 7.37k

commoncrawl/statistics

Viewer • Updated 8 days ago • 652k • 1.08k • 30

in commoncrawl/host-index-testing-v2 13 days ago

Update README.md

#1 opened 13 days ago by

updated 2 Spaces 17 days ago

Web Graph Embeddings

What is actually inside llms.txt

Analyze llms.txt files for content insights

in commoncrawl/commonlid-results 18 days ago

Adding results from GoogleTranslate for commonlid_nano

#5 opened 18 days ago by

updated a bucket 19 days ago

commoncrawl/warc2zip-examples

updated a dataset 20 days ago

commoncrawl/llms.txt

Viewer • Updated 20 days ago • 598k • 255 • 2