Buckets:
akshara-web-draft
Draft, work in progress. Intermediate data for the Akshara English + Telugu language models, built with dataflow. Contents change while the pipelines run; nothing here is a release. Released datasets will have their own dataset cards.
| Folder | What | Source and terms |
|---|---|---|
telugu-cc-warc/ |
Common Crawl WARC records whose CLD2 tags include Telugu (tel), all language-tagged crawls (CC-MAIN-2018-39 onward), fetched by byte range, records unchanged |
Common Crawl terms of use |
telugu-hplt/ |
Telugu documents (openlid-v3 tel_Telu, p ≥ 0.5, robots.txt-allowed) from the HPLT 4.0 pool, Common Crawl and Internet Archive crawls |
HPLT 4.0 terms; CC0 packaging |
english-cc-main-2025-26/ |
One Common Crawl dump through the FineWeb-style English recipe | Common Crawl terms; ODC-By for our processing |
Before any of this is used for training, it passes dataflow's policy gate (robots.txt, AI-crawler opt-outs, Content-Signal, aipref, noai, TDMRep), deduplication and decontamination.
Takedown: open an issue at https://github.com/kuluruvineeth/dataflow/issues with the URLs, and they are removed from this bucket and from every derived dataset.
Xet Storage Details
- Size:
- 1.33 kB
- Xet hash:
- 8fc8a3e752091c38a2dfdca308ad72ab7b2e772cf71167df7dd23e13110a846c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.