Rename the dataset repository to Can111/ShellElastic-data
Browse files
README.md
CHANGED
|
@@ -4,7 +4,7 @@ language:
|
|
| 4 |
- en
|
| 5 |
datasets:
|
| 6 |
- mlfoundations/dclm-baseline-1.0
|
| 7 |
-
- Can111/ShellElastic-
|
| 8 |
library_name: pytorch
|
| 9 |
pipeline_tag: text-generation
|
| 10 |
tags:
|
|
@@ -76,7 +76,7 @@ ShellElastic (Section 3.1 of the paper) organizes the hidden dimensions into a d
|
|
| 76 |
|
| 77 |
## Training
|
| 78 |
|
| 79 |
-
**Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-
|
| 80 |
|
| 81 |
**Recipe.** Trained as described in the paper (Sections 3.3 and 4.1, Appendices E.1 and E.2, and Table 9).
|
| 82 |
|
|
@@ -155,7 +155,7 @@ Apache License 2.0 (`LICENSE`).
|
|
| 155 |
Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
|
| 156 |
the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
|
| 157 |
included in this repository; the tokenized data of these runs are in the dataset repository
|
| 158 |
-
[Can111/ShellElastic-
|
| 159 |
|
| 160 |
## Citation
|
| 161 |
|
|
|
|
| 4 |
- en
|
| 5 |
datasets:
|
| 6 |
- mlfoundations/dclm-baseline-1.0
|
| 7 |
+
- Can111/ShellElastic-data
|
| 8 |
library_name: pytorch
|
| 9 |
pipeline_tag: text-generation
|
| 10 |
tags:
|
|
|
|
| 76 |
|
| 77 |
## Training
|
| 78 |
|
| 79 |
+
**Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data).
|
| 80 |
|
| 81 |
**Recipe.** Trained as described in the paper (Sections 3.3 and 4.1, Appendices E.1 and E.2, and Table 9).
|
| 82 |
|
|
|
|
| 155 |
Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
|
| 156 |
the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
|
| 157 |
included in this repository; the tokenized data of these runs are in the dataset repository
|
| 158 |
+
[Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
|
| 159 |
|
| 160 |
## Citation
|
| 161 |
|