Hugging Face's logo

hybrid-diff-ar
/
olmo2-checkpoints

Safetensors
Model card Files
xet
 Community
guanning commited on
Commit
d1d5df3
·
verified ·
1 Parent(s): 44be041

Add files using upload-large-folder tool

Browse files
Files changed (50) hide show
  1. pretrain/README.md +45 -36
  2. pretrain/olmo2-600M-10xC-k1/step100000/.metadata.json +1 -0
  3. pretrain/olmo2-600M-10xC-k1/step100000/data_paths.txt +50 -0
  4. pretrain/olmo2-600M-10xC-k1/step113000-hf/generation_config.json +11 -0
  5. pretrain/olmo2-600M-10xC-k1/step113000-hf/tokenizer_config.json +13 -0
  6. pretrain/olmo2-600M-10xC-k4/step100000-hf/chat_template.jinja +4 -0
  7. pretrain/olmo2-600M-10xC-k4/step100000-hf/config.json +29 -0
  8. pretrain/olmo2-600M-10xC-k4/step100000-hf/generation_config.json +11 -0
  9. pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer-export-audit.json +9 -0
  10. pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer.json +0 -0
  11. pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer_config.json +13 -0
  12. pretrain/olmo2-600M-10xC-k4/step100000/.metadata.json +1 -0
  13. pretrain/olmo2-600M-10xC-k4/step100000/config.json +1 -0
  14. pretrain/olmo2-600M-10xC-k4/step100000/data_paths.txt +50 -0
  15. pretrain/olmo2-600M-10xC-k4/step105000-hf/chat_template.jinja +4 -0
  16. pretrain/olmo2-600M-10xC-k4/step105000-hf/config.json +29 -0
  17. pretrain/olmo2-600M-10xC-k4/step105000-hf/generation_config.json +11 -0
  18. pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer-export-audit.json +9 -0
  19. pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer.json +0 -0
  20. pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer_config.json +13 -0
  21. pretrain/olmo2-600M-10xC-k4/step105000/.metadata.json +1 -0
  22. pretrain/olmo2-600M-10xC-k4/step105000/config.json +1 -0
  23. pretrain/olmo2-600M-10xC-k4/step105000/data_paths.txt +50 -0
  24. pretrain/olmo2-600M-10xC-k4/step110000-hf/chat_template.jinja +4 -0
  25. pretrain/olmo2-600M-10xC-k4/step110000-hf/config.json +29 -0
  26. pretrain/olmo2-600M-10xC-k4/step110000-hf/generation_config.json +11 -0
  27. pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer-export-audit.json +9 -0
  28. pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer.json +0 -0
  29. pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer_config.json +13 -0
  30. pretrain/olmo2-600M-10xC-k4/step110000/.metadata.json +1 -0
  31. pretrain/olmo2-600M-10xC-k4/step110000/config.json +1 -0
  32. pretrain/olmo2-600M-10xC-k4/step110000/data_paths.txt +50 -0
  33. pretrain/olmo2-600M-10xC-k4/step113000-hf/chat_template.jinja +4 -0
  34. pretrain/olmo2-600M-10xC-k4/step113000-hf/config.json +29 -0
  35. pretrain/olmo2-600M-10xC-k4/step113000-hf/generation_config.json +11 -0
  36. pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer-export-audit.json +9 -0
  37. pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer.json +0 -0
  38. pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer_config.json +13 -0
  39. pretrain/olmo2-600M-10xC-k4/step113000/.metadata.json +1 -0
  40. pretrain/olmo2-600M-10xC-k4/step113000/config.json +1 -0
  41. pretrain/olmo2-600M-10xC-k4/step113000/data_paths.txt +50 -0
  42. pretrain/olmo2-600M-10xC-k4/step113942-hf/chat_template.jinja +4 -0
  43. pretrain/olmo2-600M-10xC-k4/step113942-hf/config.json +29 -0
  44. pretrain/olmo2-600M-10xC-k4/step113942-hf/generation_config.json +11 -0
  45. pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer-export-audit.json +9 -0
  46. pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer.json +0 -0
  47. pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer_config.json +13 -0
  48. pretrain/olmo2-600M-10xC-k4/step113942/.metadata.json +1 -0
  49. pretrain/olmo2-600M-10xC-k4/step113942/config.json +1 -0
  50. pretrain/olmo2-600M-10xC-k4/step113942/data_paths.txt +50 -0
pretrain/README.md CHANGED
@@ -1,14 +1,14 @@
1
- # OLMo2 190M pretraining checkpoints: pass@k (k=1, k=4 and k=16)
2
 
3
- Three OLMo2-190M models pretrained from scratch with the standard OLMo2 ladder recipe for
4
- 10× Chinchilla (38.07B tokens). They are identical except for the training objective:
5
 
6
- - **k=1** (`olmo2-190M-10xC-k1`): the usual next-token cross-entropy, `-log p`.
7
- - **k=4** (`olmo2-190M-10xC-k4`): the per-token pass@k loss `-log(1 - (1-p)^k)` with k=4, where
8
  `p` is the model's probability of the correct next token.
9
- - **k=16** (`olmo2-190M-10xC-k16`): the same pass@k loss with k=16.
10
 
11
- Same data, data order, initialization and hyperparameters.
12
 
13
  ## Layout
14
 
@@ -18,14 +18,19 @@ pretrain/
18
  step72615-hf/ final model, Hugging Face transformers format (bf16 safetensors + tokenizer)
19
  step72615/ final model, OLMo-core distributed checkpoint (fp32 weights + AdamW state + trainer state)
20
  step72000[-hf]/ step70000[-hf]/ step65000[-hf]/ step60000[-hf]/ earlier checkpoints, same two formats
21
- olmo2-190M-10xC-k4/
22
- (same)
23
- olmo2-190M-10xC-k16/
24
- (same)
 
 
 
 
25
  ```
26
 
27
- Step 72615 is the end of training. Only the last checkpoints of each run were kept. Every `-hf`
28
- export was checked against its OLMo-core checkpoint: both models give the same outputs.
 
29
 
30
  ## Usage
31
 
@@ -33,7 +38,7 @@ export was checked against its OLMo-core checkpoint: both models give the same o
33
  from transformers import AutoModelForCausalLM, AutoTokenizer
34
 
35
  repo = "hybrid-diff-ar/olmo2-checkpoints"
36
- sub = "pretrain/olmo2-190M-10xC-k16/step72615-hf"
37
  tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
38
  model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub) # Olmo2ForCausalLM
39
  ```
@@ -42,34 +47,38 @@ The `stepN/` directories are OLMo-core checkpoints. Use them to resume training
42
  OLMo-core, for example by passing one as the trainer's `load_path`. Their `config.json` holds the
43
  full experiment config; the data and run paths in it refer to the cluster the models were trained on.
44
 
45
- ## Model
46
 
47
- | | |
48
- |---|---|
49
- | architecture | OLMo2 (`olmo2_190M`): 12 layers, d_model 768, 12 heads, MLP 3072 (SwiGLU), RMSNorm with QK-norm, RoPE θ=500k, untied embeddings |
50
- | parameters | 267.4M total, 190.4M non-embedding |
51
- | tokenizer | `allenai/dolma2-tokenizer` (vocab 100,278; embeddings padded to 100,352 in the OLMo-core checkpoints) |
52
- | context length | 2048 |
 
53
 
54
  ## Training
55
 
56
- | | |
57
- |---|---|
58
- | data | [`hybrid-diff-ar/olmo-data-50B`](https://huggingface.co/datasets/hybrid-diff-ar/olmo-data-50B), a source-proportional subset of OLMoE-mix-0824 (dolma2-tokenized), with the OLMo2 repetition instance filter |
59
- | tokens | 38,070,835,200 (10 × 20 × 190.4M non-embedding params), 72,615 steps |
60
- | batch | 256 × 2048 = 524,288 tokens |
61
- | optimizer | AdamW, lr 3.89e-3, betas (0.9, 0.95), weight decay 0.1 (none on embeddings), grad clip 1.0 |
62
- | schedule | cosine, 363 warmup steps, decaying to 0.1 × lr |
63
- | precision | bf16 params, fp32 reduce (HSDP); 8 × H200 for k=1 and k=16, 8 × H100 for k=4 |
64
- | seeds | init 2352, data 34521 |
65
 
66
  ## Final validation (c4_en, v3-small validation mix)
67
 
68
  | run | CE loss (= pass@1 loss) | pass@4 loss | pass@16 loss |
69
  |---|---|---|---|
70
- | k=1 | 3.339 | 2.314 | 1.527 |
71
- | k=4 | 3.380 | 2.281 | 1.465 |
72
- | k=16 | 3.546 | 2.345 | 1.427 |
73
-
74
- Training with pass@k gives up some cross-entropy and gets a lower pass@k loss in return; each model
75
- is the best of the three at the k it was trained with.
 
 
 
 
1
+ # OLMo2 pretraining checkpoints: pass@k (k=1, k=4 and k=16), 190M and 600M
2
 
3
+ OLMo2 models pretrained from scratch with the standard OLMo2 ladder recipe for 10× Chinchilla, at two
4
+ sizes. At each size the three runs are identical except for the training objective:
5
 
6
+ - **k=1** (`olmo2-<size>-10xC-k1`): the usual next-token cross-entropy, `-log p`.
7
+ - **k=4** (`olmo2-<size>-10xC-k4`): the per-token pass@k loss `-log(1 - (1-p)^k)` with k=4, where
8
  `p` is the model's probability of the correct next token.
9
+ - **k=16** (`olmo2-<size>-10xC-k16`): the same pass@k loss with k=16.
10
 
11
+ Same data, data order, initialization and hyperparameters within each size.
12
 
13
  ## Layout
14
 
 
18
  step72615-hf/ final model, Hugging Face transformers format (bf16 safetensors + tokenizer)
19
  step72615/ final model, OLMo-core distributed checkpoint (fp32 weights + AdamW state + trainer state)
20
  step72000[-hf]/ step70000[-hf]/ step65000[-hf]/ step60000[-hf]/ earlier checkpoints, same two formats
21
+ olmo2-190M-10xC-k4/ (same)
22
+ olmo2-190M-10xC-k16/ (same)
23
+ olmo2-600M-10xC-k1/
24
+ step113942-hf/ final model, Hugging Face transformers format
25
+ step113942/ final model, OLMo-core distributed checkpoint
26
+ step113000[-hf]/ step110000[-hf]/ step105000[-hf]/ step100000[-hf]/ earlier checkpoints
27
+ olmo2-600M-10xC-k4/ (same)
28
+ olmo2-600M-10xC-k16/ (same)
29
  ```
30
 
31
+ Step 72615 (190M) and step 113942 (600M) are the end of training. Only the last checkpoints of each
32
+ run were kept. Every `-hf` export was checked against its OLMo-core checkpoint: both models give the
33
+ same outputs.
34
 
35
  ## Usage
36
 
 
38
  from transformers import AutoModelForCausalLM, AutoTokenizer
39
 
40
  repo = "hybrid-diff-ar/olmo2-checkpoints"
41
+ sub = "pretrain/olmo2-600M-10xC-k16/step113942-hf"
42
  tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)
43
  model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub) # Olmo2ForCausalLM
44
  ```
 
47
  OLMo-core, for example by passing one as the trainer's `load_path`. Their `config.json` holds the
48
  full experiment config; the data and run paths in it refer to the cluster the models were trained on.
49
 
50
+ ## Models
51
 
52
+ | | 190M | 600M |
53
+ |---|---|---|
54
+ | architecture | OLMo2 (`olmo2_190M`): 12 layers, d_model 768, 12 heads, MLP 3072 | OLMo2 (`olmo2_600M`): 16 layers, d_model 1344, 16 heads, MLP 5376 |
55
+ | common | SwiGLU, RMSNorm with QK-norm, RoPE θ=500k, untied embeddings | (same) |
56
+ | parameters | 267.4M total, 190.4M non-embedding | 732.3M total, 597.4M non-embedding |
57
+ | tokenizer | `allenai/dolma2-tokenizer` (vocab 100,278; embeddings padded to 100,352 in the OLMo-core checkpoints) | (same) |
58
+ | context length | 2048 | 2048 |
59
 
60
  ## Training
61
 
62
+ | | 190M | 600M |
63
+ |---|---|---|
64
+ | data | [`hybrid-diff-ar/olmo-data-50B`](https://huggingface.co/datasets/hybrid-diff-ar/olmo-data-50B), a source-proportional subset of OLMoE-mix-0824 (dolma2-tokenized), with the OLMo2 repetition instance filter | [`hybrid-diff-ar/olmo-data-160B`](https://huggingface.co/datasets/hybrid-diff-ar/olmo-data-160B), the same kind of subset at 160B tokens, same filter |
65
+ | tokens | 38,070,835,200 (10 × 20 × 190.4M non-embedding params), 72,615 steps | 119,476,492,800 (10 × 20 × 597.4M), 113,942 steps |
66
+ | batch | 256 × 2048 = 524,288 tokens | 512 × 2048 = 1,048,576 tokens |
67
+ | optimizer | AdamW, lr 3.89e-3, betas (0.9, 0.95), weight decay 0.1 (none on embeddings), grad clip 1.0 | AdamW, lr 2.66e-3, otherwise the same |
68
+ | schedule | cosine, 363 warmup steps, decaying to 0.1 × lr | cosine, 570 warmup steps, decaying to 0.1 × lr |
69
+ | precision | bf16 params, fp32 reduce (HSDP); 8 × H200 for k=1 and k=16, 8 × H100 for k=4 | bf16 params, fp32 reduce (HSDP); 8 × H200 |
70
+ | seeds | init 2352, data 34521 | init 2352, data 34521 |
71
 
72
  ## Final validation (c4_en, v3-small validation mix)
73
 
74
  | run | CE loss (= pass@1 loss) | pass@4 loss | pass@16 loss |
75
  |---|---|---|---|
76
+ | 190M k=1 | 3.339 | 2.314 | 1.527 |
77
+ | 190M k=4 | 3.380 | 2.281 | 1.465 |
78
+ | 190M k=16 | 3.546 | 2.345 | 1.427 |
79
+ | 600M k=1 | 3.000 | 2.028 | 1.307 |
80
+ | 600M k=4 | 3.047 | 1.992 | 1.243 |
81
+ | 600M k=16 | 3.244 | 2.065 | 1.202 |
82
+
83
+ Training with pass@k gives up some cross-entropy and gets a lower pass@k loss in return; at both sizes
84
+ each model is the best of the three at the k it was trained with.
pretrain/olmo2-600M-10xC-k1/step100000/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": false, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k1/step100000/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy
pretrain/olmo2-600M-10xC-k1/step113000-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k1/step113000-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step100000-hf/chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {% for message in messages %}{{'<|im_start|>' + message['role'] + '
2
+ ' + message['content'] + '<|im_end|>' + '
3
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
4
+ ' }}{% endif %}
pretrain/olmo2-600M-10xC-k4/step100000-hf/config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Olmo2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 100257,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 100257,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1344,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 5376,
14
+ "max_position_embeddings": 2048,
15
+ "model_type": "olmo2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 16,
18
+ "num_key_value_heads": 16,
19
+ "pad_token_id": 100277,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_parameters": {
22
+ "rope_theta": 500000,
23
+ "rope_type": "default"
24
+ },
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "5.15.1",
27
+ "use_cache": true,
28
+ "vocab_size": 100278
29
+ }
pretrain/olmo2-600M-10xC-k4/step100000-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer-export-audit.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "source": "allenai/dolma2-tokenizer",
4
+ "revision": null,
5
+ "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
6
+ "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "probe_count": 8
9
+ }
pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
pretrain/olmo2-600M-10xC-k4/step100000-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step100000/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": false, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k4/step100000/config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model": {"d_model": 1344, "vocab_size": 100352, "n_layers": 16, "block": {"sequence_mixer": {"name": "default", "n_heads": 16, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "torch", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 5376, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 1048576, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.002657591869667391, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 570, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 119476492800, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-600M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-600M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}
pretrain/olmo2-600M-10xC-k4/step100000/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy
pretrain/olmo2-600M-10xC-k4/step105000-hf/chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {% for message in messages %}{{'<|im_start|>' + message['role'] + '
2
+ ' + message['content'] + '<|im_end|>' + '
3
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
4
+ ' }}{% endif %}
pretrain/olmo2-600M-10xC-k4/step105000-hf/config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Olmo2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 100257,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 100257,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1344,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 5376,
14
+ "max_position_embeddings": 2048,
15
+ "model_type": "olmo2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 16,
18
+ "num_key_value_heads": 16,
19
+ "pad_token_id": 100277,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_parameters": {
22
+ "rope_theta": 500000,
23
+ "rope_type": "default"
24
+ },
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "5.15.1",
27
+ "use_cache": true,
28
+ "vocab_size": 100278
29
+ }
pretrain/olmo2-600M-10xC-k4/step105000-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer-export-audit.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "source": "allenai/dolma2-tokenizer",
4
+ "revision": null,
5
+ "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
6
+ "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "probe_count": 8
9
+ }
pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
pretrain/olmo2-600M-10xC-k4/step105000-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step105000/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": false, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k4/step105000/config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model": {"d_model": 1344, "vocab_size": 100352, "n_layers": 16, "block": {"sequence_mixer": {"name": "default", "n_heads": 16, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "torch", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 5376, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 1048576, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.002657591869667391, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 570, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 119476492800, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-600M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-600M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}
pretrain/olmo2-600M-10xC-k4/step105000/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy
pretrain/olmo2-600M-10xC-k4/step110000-hf/chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {% for message in messages %}{{'<|im_start|>' + message['role'] + '
2
+ ' + message['content'] + '<|im_end|>' + '
3
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
4
+ ' }}{% endif %}
pretrain/olmo2-600M-10xC-k4/step110000-hf/config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Olmo2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 100257,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 100257,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1344,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 5376,
14
+ "max_position_embeddings": 2048,
15
+ "model_type": "olmo2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 16,
18
+ "num_key_value_heads": 16,
19
+ "pad_token_id": 100277,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_parameters": {
22
+ "rope_theta": 500000,
23
+ "rope_type": "default"
24
+ },
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "5.15.1",
27
+ "use_cache": true,
28
+ "vocab_size": 100278
29
+ }
pretrain/olmo2-600M-10xC-k4/step110000-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer-export-audit.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "source": "allenai/dolma2-tokenizer",
4
+ "revision": null,
5
+ "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
6
+ "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "probe_count": 8
9
+ }
pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
pretrain/olmo2-600M-10xC-k4/step110000-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step110000/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": false, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k4/step110000/config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model": {"d_model": 1344, "vocab_size": 100352, "n_layers": 16, "block": {"sequence_mixer": {"name": "default", "n_heads": 16, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "torch", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 5376, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 1048576, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.002657591869667391, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 570, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 119476492800, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-600M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-600M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}
pretrain/olmo2-600M-10xC-k4/step110000/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy
pretrain/olmo2-600M-10xC-k4/step113000-hf/chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {% for message in messages %}{{'<|im_start|>' + message['role'] + '
2
+ ' + message['content'] + '<|im_end|>' + '
3
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
4
+ ' }}{% endif %}
pretrain/olmo2-600M-10xC-k4/step113000-hf/config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Olmo2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 100257,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 100257,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1344,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 5376,
14
+ "max_position_embeddings": 2048,
15
+ "model_type": "olmo2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 16,
18
+ "num_key_value_heads": 16,
19
+ "pad_token_id": 100277,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_parameters": {
22
+ "rope_theta": 500000,
23
+ "rope_type": "default"
24
+ },
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "5.15.1",
27
+ "use_cache": true,
28
+ "vocab_size": 100278
29
+ }
pretrain/olmo2-600M-10xC-k4/step113000-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer-export-audit.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "source": "allenai/dolma2-tokenizer",
4
+ "revision": null,
5
+ "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
6
+ "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "probe_count": 8
9
+ }
pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
pretrain/olmo2-600M-10xC-k4/step113000-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step113000/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": true, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k4/step113000/config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model": {"d_model": 1344, "vocab_size": 100352, "n_layers": 16, "block": {"sequence_mixer": {"name": "default", "n_heads": 16, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "torch", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 5376, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 1048576, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.002657591869667391, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 570, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 119476492800, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-600M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-600M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}
pretrain/olmo2-600M-10xC-k4/step113000/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy
pretrain/olmo2-600M-10xC-k4/step113942-hf/chat_template.jinja ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {% for message in messages %}{{'<|im_start|>' + message['role'] + '
2
+ ' + message['content'] + '<|im_end|>' + '
3
+ '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
4
+ ' }}{% endif %}
pretrain/olmo2-600M-10xC-k4/step113942-hf/config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Olmo2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 100257,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 100257,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 1344,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 5376,
14
+ "max_position_embeddings": 2048,
15
+ "model_type": "olmo2",
16
+ "num_attention_heads": 16,
17
+ "num_hidden_layers": 16,
18
+ "num_key_value_heads": 16,
19
+ "pad_token_id": 100277,
20
+ "rms_norm_eps": 1e-06,
21
+ "rope_parameters": {
22
+ "rope_theta": 500000,
23
+ "rope_type": "default"
24
+ },
25
+ "tie_word_embeddings": false,
26
+ "transformers_version": "5.15.1",
27
+ "use_cache": true,
28
+ "vocab_size": 100278
29
+ }
pretrain/olmo2-600M-10xC-k4/step113942-hf/generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 100257,
4
+ "do_sample": true,
5
+ "eos_token_id": 100257,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 100277,
9
+ "transformers_version": "5.15.1",
10
+ "use_cache": true
11
+ }
pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer-export-audit.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "passed": true,
3
+ "source": "allenai/dolma2-tokenizer",
4
+ "revision": null,
5
+ "backend_sha256": "30ddf87d4cc948a8bd56980b84605ee465064d0b5871c7bf2fa6d00f04c99b9a",
6
+ "tokenizer_sha256": "18e309ad7f9c60037eaf26aca401a96128187cad2bdfa01a508dca7526bf6300",
7
+ "tokenizer_class": "TokenizersBackend",
8
+ "probe_count": 8
9
+ }
pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
pretrain/olmo2-600M-10xC-k4/step113942-hf/tokenizer_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": "<|endoftext|>",
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|endoftext|>",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_max_length": 2048,
10
+ "pad_token": "<|pad|>",
11
+ "tokenizer_class": "TokenizersBackend",
12
+ "unk_token": "<|endoftext|>"
13
+ }
pretrain/olmo2-600M-10xC-k4/step113942/.metadata.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ephemeral": false, "version": "3.0.0"}
pretrain/olmo2-600M-10xC-k4/step113942/config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"model": {"d_model": 1344, "vocab_size": 100352, "n_layers": 16, "block": {"sequence_mixer": {"name": "default", "n_heads": 16, "bias": false, "rope": {"name": "default", "theta": 500000, "full_precision": true, "no_global_rope": false, "partial_rotary_factor": 1.0, "_CLASS_": "olmo_core.nn.rope.RoPEConfig"}, "qk_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "backend": "torch", "dtype": "float32", "use_head_qk_norm": false, "scalable_softmax": false, "attention_sinks": false, "_CLASS_": "olmo_core.nn.attention.AttentionConfig", "type": "attention"}, "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "feed_forward": {"hidden_size": 5376, "name": "default", "bias": false, "dtype": "float32", "activation": "silu", "_CLASS_": "olmo_core.nn.feed_forward.FeedForwardConfig"}, "name": "reordered_norm", "_CLASS_": "olmo_core.nn.transformer.config.TransformerBlockConfig"}, "lm_head": {"name": "default", "layer_norm": {"name": "rms", "eps": 1e-06, "bias": false, "dtype": "float32", "_CLASS_": "olmo_core.nn.layer_norm.LayerNormConfig"}, "bias": false, "dtype": "float32", "loss_implementation": "default", "_CLASS_": "olmo_core.nn.lm_head.LMHeadConfig"}, "name": "default", "dtype": "float32", "init_method": "normal", "init_seed": 2352, "init_std": 0.02, "tie_word_embeddings": false, "_CLASS_": "olmo_core.nn.transformer.config.TransformerConfig"}, "dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "paths": ["/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy", "/data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy"], "expand_glob": false, "metadata": [{"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "dclm"}, {"label": "pes2o"}, {"label": "proofpile-2-arxiv"}, {"label": "proofpile-2-open-web-math"}, {"label": "proofpile-2-stack"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "starcoder"}, {"label": "wikipedia"}], "include_instance_metadata": true, "instance_filter_config": {"repetition_max_period": 13, "repetition_min_period": 1, "repetition_max_count": 32, "_CLASS_": "olmo_core.data.numpy_dataset.InstanceFilterConfig"}, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "generate_doc_lengths": false, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyFSLDatasetConfig"}, "data_loader": {"global_batch_size": 1048576, "seed": 34521, "num_workers": 4, "ignore_fingerprint_mismatch": false, "_CLASS_": "olmo_core.data.data_loader.NumpyDataLoaderConfig", "type": "numpy"}, "train_module": {"rank_microbatch_size": 65536, "max_sequence_length": 2048, "optim": {"group_overrides": [{"params": ["embeddings.weight"], "opts": {"weight_decay": 0.0}, "_CLASS_": "olmo_core.optim.config.OptimGroupOverride"}], "compile": false, "fixed_fields": ["initial_lr"], "lr": 0.002657591869667391, "betas": [0.9, 0.95], "eps": 1e-08, "weight_decay": 0.1, "fused": true, "_CLASS_": "olmo_core.optim.adamw.AdamWConfig", "type": "adamw"}, "max_grad_norm": 1.0, "scheduler": {"lr_field": "lr", "initial_lr_field": "initial_lr", "units": "steps", "warmup": 570, "alpha_f": 0.1, "warmup_min_lr": 0.0, "_CLASS_": "olmo_core.optim.scheduler.CosWithWarmup", "type": "cos_with_warmup"}, "compile_model": true, "dp_config": {"name": "hsdp", "param_dtype": "bfloat16", "reduce_dtype": "float32", "wrapping_strategy": "full", "prefetch_factor": 0, "only_allreduce_last_microbatch": true, "reduce_grads_in_fp32": true, "accumulate_grads_in_fp32": true, "use_reduce_scatter": false, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerDataParallelConfig"}, "passk_k": 4, "passk_log_top": 65536, "label_ignore_index": -100, "_CLASS_": "olmo_core.train.train_module.transformer.config.TransformerTrainModuleConfig"}, "trainer": {"save_folder": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/checkpoints", "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "load_strategy": "if_available", "checkpointer": {"pre_download": false, "throttle_uploads": false, "_CLASS_": "olmo_core.train.checkpoint.CheckpointerConfig"}, "save_overwrite": false, "max_duration": {"value": 119476492800, "unit": "tokens", "_CLASS_": "olmo_core.train.common.Duration"}, "cancel_check_interval": 10, "metrics_collect_interval": 10, "callbacks": {"gpu_monitor": {"_CLASS_": "olmo_core.train.callbacks.gpu_memory_monitor.GPUMemoryMonitorCallback"}, "config_saver": {"fname": "config.json", "_CLASS_": "olmo_core.train.callbacks.config_saver.ConfigSaverCallback"}, "garbage_collector": {"gc_interval": 1000, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.garbage_collector.GarbageCollectorCallback"}, "lm_evaluator": {"eval_dataset": {"tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "mix": "v3-small-ppl-validation", "mix_base_dir": "/data/home/guanning/olmo_data/olmo-data.org", "expand_glob": false, "include_instance_metadata": true, "work_dir": "/data/home/guanning/olmo_runs/olmo2-600M-10xC-k4/work", "ignore_fingerprint_mismatch": false, "sequence_length": 2048, "_CLASS_": "olmo_core.data.numpy_dataset.NumpyPaddedFSLDatasetConfig"}, "eval_interval": 2000, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "log_interval": 5, "deterministic": true, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.LMEvaluatorCallbackConfig"}, "downstream_evaluator": {"tasks": ["arc_challenge_test_bpb_5shot", "arc_challenge_test_mc_5shot_fast", "arc_easy_test_bpb_5shot", "arc_easy_test_mc_5shot_fast", "basic_skills_arithmetic_rc_5shot", "basic_skills_coding_rc_5shot", "basic_skills_common_knowledge_rc_5shot", "basic_skills_logical_reasoning_rc_5shot", "basic_skills_pattern_rc_5shot", "basic_skills_string_operations_rc_5shot", "codex_humaneval_gold_bpb_3shot", "codex_mbpp_gold_bpb_3shot", "copycolors_10way_fast", "hellaswag_bpb_5shot", "minerva_math_500_gold_bpb_0shot", "mmlu_humanities_test_bpb_5shot", "mmlu_humanities_test_mc_5shot_fast", "mmlu_other_test_bpb_5shot", "mmlu_other_test_mc_5shot_fast", "mmlu_social_sciences_test_bpb_5shot", "mmlu_social_sciences_test_mc_5shot_fast", "mmlu_stem_test_bpb_5shot", "mmlu_stem_test_mc_5shot_fast", "mt_mbpp_cpp_gold_bpb_3shot", "mt_mbpp_java_gold_bpb_3shot", "mt_mbpp_rust_gold_bpb_3shot"], "tokenizer": {"vocab_size": 100278, "eos_token_id": 100257, "pad_token_id": 100277, "identifier": "allenai/dolma2-tokenizer", "_CLASS_": "olmo_core.data.tokenizer.TokenizerConfig"}, "eval_interval": 10000, "eval_duration": {"value": 1, "unit": "epochs", "_CLASS_": "olmo_core.train.common.Duration"}, "eval_on_startup": false, "eval_on_finish": true, "cancel_after_first_eval": false, "log_interval": 5, "lazy": false, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.evaluator_callback.DownstreamEvaluatorCallbackConfig"}, "checkpointer": {"save_interval": 5000, "ephemeral_save_interval": 1000, "save_async": true, "remove": "ephemeral_only", "max_checkpoints": 3, "enabled": true, "_CLASS_": "olmo_core.train.callbacks.checkpointer.CheckpointerCallback"}, "wandb": {"enabled": true, "name": "olmo2-600M-10xC-k4", "project": "olmo-pretrain-1018", "entity": "train-llm", "group": "olmo2-600M-10xC", "cancel_tags": ["cancel", "canceled", "cancelled"], "cancel_check_interval": 10, "_CLASS_": "olmo_core.train.callbacks.wandb.WandBCallback"}}, "bookkeeping_soft_timeout": 30, "no_checkpoints": false, "no_evals": false, "_CLASS_": "olmo_core.train.config.TrainerConfig"}, "init_seed": 2352, "_CLASS_": "__main__.ExperimentConfig"}
pretrain/olmo2-600M-10xC-k4/step113942/data_paths.txt ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-154-00004.npy
2
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-037-00000.npy
3
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-073-00003.npy
4
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-043-00001.npy
5
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-058-00004.npy
6
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00001.npy
7
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-118-00004.npy
8
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00000.npy
9
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-186-00004.npy
10
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-014-00004.npy
11
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-036-00002.npy
12
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00003.npy
13
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-160-00004.npy
14
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-090-00003.npy
15
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-030-00002.npy
16
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-069-00002.npy
17
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-176-00002.npy
18
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-140-00001.npy
19
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-162-00001.npy
20
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-027-00001.npy
21
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-130-00003.npy
22
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-008-00001.npy
23
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-124-00001.npy
24
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-125-00001.npy
25
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-117-00002.npy
26
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-152-00002.npy
27
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-016-00001.npy
28
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00001.npy
29
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-107-00000.npy
30
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-110-00004.npy
31
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-071-00002.npy
32
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-170-00003.npy
33
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-105-00003.npy
34
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-114-00004.npy
35
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-135-00003.npy
36
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-174-00002.npy
37
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-024-00004.npy
38
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-078-00001.npy
39
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-031-00003.npy
40
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/dclm/text_openhermes_reddit_eli5_vs_rw_v2_bigram_200k_train/allenai/dolma2-tokenizer/part-171-00000.prefix6308545764.npy
41
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/pes2o/allenai/dolma2-tokenizer/part-09-00000.prefix9671333560.npy
42
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/arxiv/train/allenai/dolma2-tokenizer/part-12-00000.prefix3431295592.npy
43
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/open-web-math/train/allenai/dolma2-tokenizer/part-09-00000.prefix2012701596.npy
44
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/proof-pile-2/v0_decontaminated/algebraic-stack/train/allenai/dolma2-tokenizer/part-05-00000.prefix1952181972.npy
45
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-009-00000.npy
46
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-065-00000.npy
47
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-038-00000.npy
48
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-087-00000.npy
49
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/starcoder/v1-decon-100_to_20k-2star-top_token_030/allenai/dolma2-tokenizer/part-011-00000.prefix1990182664.npy
50
+ /data/home/guanning/olmo_data/olmo-data.org/preprocessed/olmo-mix/danyh-compiled-v1_7/documents/wiki/allenai/dolma2-tokenizer/part-1-00000.prefix604873468.npy