Buckets:
394 GB
80,053 files
Updated 4 days ago
Ctrl+K
| Name | Size | Uploaded | Xet hash |
|---|---|---|---|
| english-cc-main-2025-26 | 57,244 items | ||
| telugu-cc-warc | 3,013 items | ||
| telugu-hplt | 19,795 items | ||
| README.md | 1.33 kB xet | 8fc8a3e7 |
akshara-web-draft
Draft, work in progress. Intermediate data for the Akshara English + Telugu language models, built with dataflow. Contents change while the pipelines run; nothing here is a release. Released datasets will have their own dataset cards.
| Folder | What | Source and terms |
|---|---|---|
telugu-cc-warc/ |
Common Crawl WARC records whose CLD2 tags include Telugu (tel), all language-tagged crawls (CC-MAIN-2018-39 onward), fetched by byte range, records unchanged |
Common Crawl terms of use |
telugu-hplt/ |
Telugu documents (openlid-v3 tel_Telu, p ≥ 0.5, robots.txt-allowed) from the HPLT 4.0 pool, Common Crawl and Internet Archive crawls |
HPLT 4.0 terms; CC0 packaging |
english-cc-main-2025-26/ |
One Common Crawl dump through the FineWeb-style English recipe | Common Crawl terms; ODC-By for our processing |
Before any of this is used for training, it passes dataflow's policy gate (robots.txt, AI-crawler opt-outs, Content-Signal, aipref, noai, TDMRep), deduplication and decontamination.
Takedown: open an issue at https://github.com/kuluruvineeth/dataflow/issues with the URLs, and they are removed from this bucket and from every derived dataset.
- Total size
- 394 GB
- Files
- 80,053
- Last updated
- Oct 23
- Pre-warmed CDN
- US EU US EU