Buckets:
| # akshara-web-draft | |
| **Draft, work in progress.** Intermediate data for the Akshara English + Telugu language models, built with | |
| [dataflow](https://github.com/kuluruvineeth/dataflow). Contents change while the pipelines run; nothing here is a | |
| release. Released datasets will have their own dataset cards. | |
| | Folder | What | Source and terms | | |
| |---|---|---| | |
| | `telugu-cc-warc/` | Common Crawl WARC records whose CLD2 tags include Telugu (`tel`), all language-tagged crawls (CC-MAIN-2018-39 onward), fetched by byte range, records unchanged | [Common Crawl terms of use](https://commoncrawl.org/terms-of-use) | | |
| | `telugu-hplt/` | Telugu documents (openlid-v3 `tel_Telu`, p ≥ 0.5, robots.txt-allowed) from the HPLT 4.0 pool, Common Crawl and Internet Archive crawls | [HPLT 4.0](https://hplt-project.org/datasets/v4.0) terms; CC0 packaging | | |
| | `english-cc-main-2025-26/` | One Common Crawl dump through the FineWeb-style English recipe | Common Crawl terms; ODC-By for our processing | | |
| Before any of this is used for training, it passes dataflow's policy gate (robots.txt, AI-crawler opt-outs, | |
| Content-Signal, aipref, noai, TDMRep), deduplication and decontamination. | |
| Takedown: open an issue at https://github.com/kuluruvineeth/dataflow/issues with the URLs, and they are removed from | |
| this bucket and from every derived dataset. | |
Xet Storage Details
- Size:
- 1.33 kB
- Xet hash:
- 8fc8a3e752091c38a2dfdca308ad72ab7b2e772cf71167df7dd23e13110a846c
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.