Can111 commited on
Commit
00ffd5d
·
verified ·
1 Parent(s): 962a123

Rename the dataset repository to Can111/ShellElastic-data

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -4,7 +4,7 @@ language:
4
  - en
5
  datasets:
6
  - mlfoundations/dclm-baseline-1.0
7
- - Can111/ShellElastic-pretraining-data
8
  library_name: pytorch
9
  pipeline_tag: text-generation
10
  tags:
@@ -76,7 +76,7 @@ ShellElastic (Section 3.1 of the paper) organizes the hidden dimensions into a d
76
 
77
  ## Training
78
 
79
- **Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-pretraining-data](https://huggingface.co/datasets/Can111/ShellElastic-pretraining-data).
80
 
81
  **Recipe.** Trained as described in the paper (Sections 3.3 and 4.1, Appendices E.1 and E.2, and Table 9).
82
 
@@ -155,7 +155,7 @@ Apache License 2.0 (`LICENSE`).
155
  Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
156
  the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
157
  included in this repository; the tokenized data of these runs are in the dataset repository
158
- [Can111/ShellElastic-pretraining-data](https://huggingface.co/datasets/Can111/ShellElastic-pretraining-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
159
 
160
  ## Citation
161
 
 
4
  - en
5
  datasets:
6
  - mlfoundations/dclm-baseline-1.0
7
+ - Can111/ShellElastic-data
8
  library_name: pytorch
9
  pipeline_tag: text-generation
10
  tags:
 
76
 
77
  ## Training
78
 
79
+ **Data.** [DCLM-Baseline](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) (Li et al., 2024), a filtered Common Crawl corpus, with the [GPT-NeoX-20B tokenizer](https://huggingface.co/EleutherAI/gpt-neox-20b) (Black et al., 2022) and a sequence length of 2,048 (Section 4.1). A run at this scale takes 25,000 steps of 128 sequences, 6.55B tokens (Section 4.1). The tokenized training and validation data of these runs are in the dataset repository [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data).
80
 
81
  **Recipe.** Trained as described in the paper (Sections 3.3 and 4.1, Appendices E.1 and E.2, and Table 9).
82
 
 
155
  Third-party terms: the training data, DCLM-Baseline, are distributed under CC-BY-4.0 and derived from Common Crawl;
156
  the validation text is from C4, distributed under ODC-BY together with the Common Crawl terms of use. Neither is
157
  included in this repository; the tokenized data of these runs are in the dataset repository
158
+ [Can111/ShellElastic-data](https://huggingface.co/datasets/Can111/ShellElastic-data). The GPT-NeoX-20B tokenizer (Apache-2.0) is not bundled here.
159
 
160
  ## Citation
161