Hugging Face's logo

hardlyworking
/
Beef-12B

PEFT
Safetensors
mistral
axolotl
Generated from Trainer
Model card Files
xet
Community
hardlyworking commited on
Commit
3da12d3
·
verified ·
1 Parent(s): 3681387

Training in progress, step 537, checkpoint

Browse files
Files changed (23) hide show
  1. checkpoint-537/README.md +202 -0
  2. checkpoint-537/adapter_config.json +42 -0
  3. checkpoint-537/adapter_model.safetensors +3 -0
  4. checkpoint-537/global_step536/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt +3 -0
  5. checkpoint-537/global_step536/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt +3 -0
  6. checkpoint-537/global_step536/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt +3 -0
  7. checkpoint-537/global_step536/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt +3 -0
  8. checkpoint-537/global_step536/zero_pp_rank_0_mp_rank_00_model_states.pt +3 -0
  9. checkpoint-537/global_step536/zero_pp_rank_1_mp_rank_00_model_states.pt +3 -0
  10. checkpoint-537/global_step536/zero_pp_rank_2_mp_rank_00_model_states.pt +3 -0
  11. checkpoint-537/global_step536/zero_pp_rank_3_mp_rank_00_model_states.pt +3 -0
  12. checkpoint-537/latest +1 -0
  13. checkpoint-537/rng_state_0.pth +3 -0
  14. checkpoint-537/rng_state_1.pth +3 -0
  15. checkpoint-537/rng_state_2.pth +3 -0
  16. checkpoint-537/rng_state_3.pth +3 -0
  17. checkpoint-537/scheduler.pt +3 -0
  18. checkpoint-537/special_tokens_map.json +30 -0
  19. checkpoint-537/tokenizer.json +3 -0
  20. checkpoint-537/tokenizer_config.json +0 -0
  21. checkpoint-537/trainer_state.json +3889 -0
  22. checkpoint-537/training_args.bin +3 -0
  23. checkpoint-537/zero_to_fp32.py +674 -0
checkpoint-537/README.md ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: hardlyworking/Noodles-Merge-12B
3
+ library_name: peft
4
+ ---
5
+
6
+ # Model Card for Model ID
7
+
8
+ <!-- Provide a quick summary of what the model is/does. -->
9
+
10
+
11
+
12
+ ## Model Details
13
+
14
+ ### Model Description
15
+
16
+ <!-- Provide a longer summary of what this model is. -->
17
+
18
+
19
+
20
+ - **Developed by:** [More Information Needed]
21
+ - **Funded by [optional]:** [More Information Needed]
22
+ - **Shared by [optional]:** [More Information Needed]
23
+ - **Model type:** [More Information Needed]
24
+ - **Language(s) (NLP):** [More Information Needed]
25
+ - **License:** [More Information Needed]
26
+ - **Finetuned from model [optional]:** [More Information Needed]
27
+
28
+ ### Model Sources [optional]
29
+
30
+ <!-- Provide the basic links for the model. -->
31
+
32
+ - **Repository:** [More Information Needed]
33
+ - **Paper [optional]:** [More Information Needed]
34
+ - **Demo [optional]:** [More Information Needed]
35
+
36
+ ## Uses
37
+
38
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
39
+
40
+ ### Direct Use
41
+
42
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
+
44
+ [More Information Needed]
45
+
46
+ ### Downstream Use [optional]
47
+
48
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
49
+
50
+ [More Information Needed]
51
+
52
+ ### Out-of-Scope Use
53
+
54
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
+
56
+ [More Information Needed]
57
+
58
+ ## Bias, Risks, and Limitations
59
+
60
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
+
62
+ [More Information Needed]
63
+
64
+ ### Recommendations
65
+
66
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
67
+
68
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
+
70
+ ## How to Get Started with the Model
71
+
72
+ Use the code below to get started with the model.
73
+
74
+ [More Information Needed]
75
+
76
+ ## Training Details
77
+
78
+ ### Training Data
79
+
80
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
+
82
+ [More Information Needed]
83
+
84
+ ### Training Procedure
85
+
86
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
+
88
+ #### Preprocessing [optional]
89
+
90
+ [More Information Needed]
91
+
92
+
93
+ #### Training Hyperparameters
94
+
95
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
+
97
+ #### Speeds, Sizes, Times [optional]
98
+
99
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
+
101
+ [More Information Needed]
102
+
103
+ ## Evaluation
104
+
105
+ <!-- This section describes the evaluation protocols and provides the results. -->
106
+
107
+ ### Testing Data, Factors & Metrics
108
+
109
+ #### Testing Data
110
+
111
+ <!-- This should link to a Dataset Card if possible. -->
112
+
113
+ [More Information Needed]
114
+
115
+ #### Factors
116
+
117
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
+
119
+ [More Information Needed]
120
+
121
+ #### Metrics
122
+
123
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
+
125
+ [More Information Needed]
126
+
127
+ ### Results
128
+
129
+ [More Information Needed]
130
+
131
+ #### Summary
132
+
133
+
134
+
135
+ ## Model Examination [optional]
136
+
137
+ <!-- Relevant interpretability work for the model goes here -->
138
+
139
+ [More Information Needed]
140
+
141
+ ## Environmental Impact
142
+
143
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
+
145
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
+
147
+ - **Hardware Type:** [More Information Needed]
148
+ - **Hours used:** [More Information Needed]
149
+ - **Cloud Provider:** [More Information Needed]
150
+ - **Compute Region:** [More Information Needed]
151
+ - **Carbon Emitted:** [More Information Needed]
152
+
153
+ ## Technical Specifications [optional]
154
+
155
+ ### Model Architecture and Objective
156
+
157
+ [More Information Needed]
158
+
159
+ ### Compute Infrastructure
160
+
161
+ [More Information Needed]
162
+
163
+ #### Hardware
164
+
165
+ [More Information Needed]
166
+
167
+ #### Software
168
+
169
+ [More Information Needed]
170
+
171
+ ## Citation [optional]
172
+
173
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
+
175
+ **BibTeX:**
176
+
177
+ [More Information Needed]
178
+
179
+ **APA:**
180
+
181
+ [More Information Needed]
182
+
183
+ ## Glossary [optional]
184
+
185
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
+
187
+ [More Information Needed]
188
+
189
+ ## More Information [optional]
190
+
191
+ [More Information Needed]
192
+
193
+ ## Model Card Authors [optional]
194
+
195
+ [More Information Needed]
196
+
197
+ ## Model Card Contact
198
+
199
+ [More Information Needed]
200
+ ### Framework versions
201
+
202
+ - PEFT 0.15.1
checkpoint-537/adapter_config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "hardlyworking/Noodles-Merge-12B",
5
+ "bias": "none",
6
+ "corda_config": null,
7
+ "eva_config": null,
8
+ "exclude_modules": null,
9
+ "fan_in_fan_out": null,
10
+ "inference_mode": true,
11
+ "init_lora_weights": true,
12
+ "layer_replication": null,
13
+ "layers_pattern": null,
14
+ "layers_to_transform": null,
15
+ "loftq_config": {},
16
+ "lora_alpha": 16,
17
+ "lora_bias": false,
18
+ "lora_dropout": 0.05,
19
+ "megatron_config": null,
20
+ "megatron_core": "megatron.core",
21
+ "modules_to_save": [
22
+ "embed_tokens",
23
+ "lm_head"
24
+ ],
25
+ "peft_type": "LORA",
26
+ "r": 128,
27
+ "rank_pattern": {},
28
+ "revision": null,
29
+ "target_modules": [
30
+ "o_proj",
31
+ "down_proj",
32
+ "gate_proj",
33
+ "up_proj",
34
+ "k_proj",
35
+ "q_proj",
36
+ "v_proj"
37
+ ],
38
+ "task_type": "CAUSAL_LM",
39
+ "trainable_token_indices": null,
40
+ "use_dora": false,
41
+ "use_rslora": true
42
+ }
checkpoint-537/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4c2b3bdb938a5080aca2a37a024dcb6c652ec05fd000e0447e5353178e0b4204
3
+ size 3596694008
checkpoint-537/global_step536/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0266aa7084d31e86a46f2ed56f2bc7b4ba3ad061890caf7e246ad513fb7c8df
3
+ size 2712460238
checkpoint-537/global_step536/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f75deeea96f26a4382ad09e3f4297fbe32731eb37404ca86c8d1c5b40f0beae5
3
+ size 2712460238
checkpoint-537/global_step536/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:418b65934f15e13b7242d782be5bf39a7f0835c5497363942d8c0843e386c190
3
+ size 2712460238
checkpoint-537/global_step536/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:913611695f1bf834b03c3c51b24e75645b948dfb02b4edb8e76825dc9fe23a79
3
+ size 2712460238
checkpoint-537/global_step536/zero_pp_rank_0_mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8535ffdaee356b6a0a44ec445b4bc4b61344f55d08a2a2cfc991cec4a5d0b2be
3
+ size 558360
checkpoint-537/global_step536/zero_pp_rank_1_mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a0d1fdd8e7cf14fb20d2c7eb5a436858ac1af70fb718c145a0a0571dd1bfcd4d
3
+ size 558360
checkpoint-537/global_step536/zero_pp_rank_2_mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ea96919503037b6ee35e95aa3cbd91ff50615f6859a0a8daa32d989443cfea71
3
+ size 558360
checkpoint-537/global_step536/zero_pp_rank_3_mp_rank_00_model_states.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4022f8c4ec8c8d3f42d121e033d50adc3e300684cee142f19da51f62080d0495
3
+ size 558360
checkpoint-537/latest ADDED
@@ -0,0 +1 @@
 
 
1
+ global_step536
checkpoint-537/rng_state_0.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ffb0bab659b725e6bd3892db991820811cfb5ee74a15421aa01d30a189f31f8b
3
+ size 15024
checkpoint-537/rng_state_1.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:56296bf6290d6c9ead324a42eb1210df2347e1a607ba1fce139e7f496656e668
3
+ size 15024
checkpoint-537/rng_state_2.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba210a4f17dce49e4c4f831b27d74f8844eac231c65bd5456652b45074b7d139
3
+ size 15024
checkpoint-537/rng_state_3.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0eaae1538cc7680ee1d1fce13ce15d398ffaf7c9d226b7d902ece8d0b95e8a0d
3
+ size 15024
checkpoint-537/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:77a4880fca3037f16bd6f56755531c564670eb15b9900814a53fad1f0741f15f
3
+ size 1064
checkpoint-537/special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<s>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|im_end|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<pad>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<unk>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
checkpoint-537/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4b4c8fcd33487a449c07f423d47adb035bba8347ccf13eb074b4d1fef8acf919
3
+ size 17078288
checkpoint-537/tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff
 
checkpoint-537/trainer_state.json ADDED
@@ -0,0 +1,3889 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.496513249651325,
6
+ "eval_steps": 45,
7
+ "global_step": 537,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.002789400278940028,
14
+ "grad_norm": 2.008944315843832,
15
+ "learning_rate": 0.0,
16
+ "loss": 1.7653,
17
+ "step": 1
18
+ },
19
+ {
20
+ "epoch": 0.002789400278940028,
21
+ "eval_loss": 1.7864996194839478,
22
+ "eval_runtime": 155.6922,
23
+ "eval_samples_per_second": 2.042,
24
+ "eval_steps_per_second": 0.128,
25
+ "step": 1
26
+ },
27
+ {
28
+ "epoch": 0.005578800557880056,
29
+ "grad_norm": 1.5986956798562522,
30
+ "learning_rate": 5.000000000000001e-07,
31
+ "loss": 1.3721,
32
+ "step": 2
33
+ },
34
+ {
35
+ "epoch": 0.008368200836820083,
36
+ "grad_norm": 2.487093074085383,
37
+ "learning_rate": 1.0000000000000002e-06,
38
+ "loss": 1.4468,
39
+ "step": 3
40
+ },
41
+ {
42
+ "epoch": 0.011157601115760111,
43
+ "grad_norm": 1.9408908369837041,
44
+ "learning_rate": 1.5e-06,
45
+ "loss": 1.8555,
46
+ "step": 4
47
+ },
48
+ {
49
+ "epoch": 0.01394700139470014,
50
+ "grad_norm": 1.855397723025267,
51
+ "learning_rate": 2.0000000000000003e-06,
52
+ "loss": 1.5173,
53
+ "step": 5
54
+ },
55
+ {
56
+ "epoch": 0.016736401673640166,
57
+ "grad_norm": 1.6798170163537172,
58
+ "learning_rate": 2.5e-06,
59
+ "loss": 1.3812,
60
+ "step": 6
61
+ },
62
+ {
63
+ "epoch": 0.019525801952580194,
64
+ "grad_norm": 2.0478066370088133,
65
+ "learning_rate": 3e-06,
66
+ "loss": 1.475,
67
+ "step": 7
68
+ },
69
+ {
70
+ "epoch": 0.022315202231520222,
71
+ "grad_norm": 1.6541684745930123,
72
+ "learning_rate": 3.5e-06,
73
+ "loss": 1.6239,
74
+ "step": 8
75
+ },
76
+ {
77
+ "epoch": 0.02510460251046025,
78
+ "grad_norm": 1.5940373225900581,
79
+ "learning_rate": 4.000000000000001e-06,
80
+ "loss": 1.4474,
81
+ "step": 9
82
+ },
83
+ {
84
+ "epoch": 0.02789400278940028,
85
+ "grad_norm": 1.3436762031717087,
86
+ "learning_rate": 4.5e-06,
87
+ "loss": 1.5277,
88
+ "step": 10
89
+ },
90
+ {
91
+ "epoch": 0.030683403068340307,
92
+ "grad_norm": 1.4820988105255548,
93
+ "learning_rate": 5e-06,
94
+ "loss": 1.3905,
95
+ "step": 11
96
+ },
97
+ {
98
+ "epoch": 0.03347280334728033,
99
+ "grad_norm": 1.3026823080341792,
100
+ "learning_rate": 5.500000000000001e-06,
101
+ "loss": 1.2716,
102
+ "step": 12
103
+ },
104
+ {
105
+ "epoch": 0.03626220362622036,
106
+ "grad_norm": 1.2786618249049673,
107
+ "learning_rate": 6e-06,
108
+ "loss": 1.4357,
109
+ "step": 13
110
+ },
111
+ {
112
+ "epoch": 0.03905160390516039,
113
+ "grad_norm": 1.4401448629219828,
114
+ "learning_rate": 6.5000000000000004e-06,
115
+ "loss": 1.5418,
116
+ "step": 14
117
+ },
118
+ {
119
+ "epoch": 0.04184100418410042,
120
+ "grad_norm": 1.4108691585319235,
121
+ "learning_rate": 7e-06,
122
+ "loss": 1.419,
123
+ "step": 15
124
+ },
125
+ {
126
+ "epoch": 0.044630404463040445,
127
+ "grad_norm": 1.434665713671276,
128
+ "learning_rate": 7.500000000000001e-06,
129
+ "loss": 1.4042,
130
+ "step": 16
131
+ },
132
+ {
133
+ "epoch": 0.04741980474198047,
134
+ "grad_norm": 1.4864898541989087,
135
+ "learning_rate": 8.000000000000001e-06,
136
+ "loss": 1.4338,
137
+ "step": 17
138
+ },
139
+ {
140
+ "epoch": 0.0502092050209205,
141
+ "grad_norm": 1.7497952919365851,
142
+ "learning_rate": 8.5e-06,
143
+ "loss": 1.5988,
144
+ "step": 18
145
+ },
146
+ {
147
+ "epoch": 0.05299860529986053,
148
+ "grad_norm": 1.5825124800137,
149
+ "learning_rate": 9e-06,
150
+ "loss": 1.4817,
151
+ "step": 19
152
+ },
153
+ {
154
+ "epoch": 0.05578800557880056,
155
+ "grad_norm": 1.5639582937511203,
156
+ "learning_rate": 9.5e-06,
157
+ "loss": 1.3795,
158
+ "step": 20
159
+ },
160
+ {
161
+ "epoch": 0.058577405857740586,
162
+ "grad_norm": 1.5512627729747626,
163
+ "learning_rate": 1e-05,
164
+ "loss": 1.4833,
165
+ "step": 21
166
+ },
167
+ {
168
+ "epoch": 0.061366806136680614,
169
+ "grad_norm": 1.4462678614801439,
170
+ "learning_rate": 1.0500000000000001e-05,
171
+ "loss": 1.3781,
172
+ "step": 22
173
+ },
174
+ {
175
+ "epoch": 0.06415620641562064,
176
+ "grad_norm": 1.5995776178618928,
177
+ "learning_rate": 1.1000000000000001e-05,
178
+ "loss": 1.4862,
179
+ "step": 23
180
+ },
181
+ {
182
+ "epoch": 0.06694560669456066,
183
+ "grad_norm": 1.2623925872690254,
184
+ "learning_rate": 1.15e-05,
185
+ "loss": 1.4222,
186
+ "step": 24
187
+ },
188
+ {
189
+ "epoch": 0.0697350069735007,
190
+ "grad_norm": 1.2816346894508914,
191
+ "learning_rate": 1.2e-05,
192
+ "loss": 1.3473,
193
+ "step": 25
194
+ },
195
+ {
196
+ "epoch": 0.07252440725244072,
197
+ "grad_norm": 1.4528035335148701,
198
+ "learning_rate": 1.25e-05,
199
+ "loss": 1.5278,
200
+ "step": 26
201
+ },
202
+ {
203
+ "epoch": 0.07531380753138076,
204
+ "grad_norm": 1.3415339044507064,
205
+ "learning_rate": 1.3000000000000001e-05,
206
+ "loss": 1.3939,
207
+ "step": 27
208
+ },
209
+ {
210
+ "epoch": 0.07810320781032078,
211
+ "grad_norm": 1.6072610009319113,
212
+ "learning_rate": 1.3500000000000001e-05,
213
+ "loss": 1.5401,
214
+ "step": 28
215
+ },
216
+ {
217
+ "epoch": 0.08089260808926081,
218
+ "grad_norm": 1.5988215139386146,
219
+ "learning_rate": 1.4e-05,
220
+ "loss": 1.3995,
221
+ "step": 29
222
+ },
223
+ {
224
+ "epoch": 0.08368200836820083,
225
+ "grad_norm": 1.4507828208303803,
226
+ "learning_rate": 1.45e-05,
227
+ "loss": 1.4584,
228
+ "step": 30
229
+ },
230
+ {
231
+ "epoch": 0.08647140864714087,
232
+ "grad_norm": 1.4975002404633069,
233
+ "learning_rate": 1.5000000000000002e-05,
234
+ "loss": 1.3834,
235
+ "step": 31
236
+ },
237
+ {
238
+ "epoch": 0.08926080892608089,
239
+ "grad_norm": 1.4128570529331854,
240
+ "learning_rate": 1.55e-05,
241
+ "loss": 1.4216,
242
+ "step": 32
243
+ },
244
+ {
245
+ "epoch": 0.09205020920502092,
246
+ "grad_norm": 1.3616495178863832,
247
+ "learning_rate": 1.6000000000000003e-05,
248
+ "loss": 1.5167,
249
+ "step": 33
250
+ },
251
+ {
252
+ "epoch": 0.09483960948396095,
253
+ "grad_norm": 1.1988790310990909,
254
+ "learning_rate": 1.65e-05,
255
+ "loss": 1.4525,
256
+ "step": 34
257
+ },
258
+ {
259
+ "epoch": 0.09762900976290098,
260
+ "grad_norm": 1.4483022749153813,
261
+ "learning_rate": 1.7e-05,
262
+ "loss": 1.5741,
263
+ "step": 35
264
+ },
265
+ {
266
+ "epoch": 0.100418410041841,
267
+ "grad_norm": 1.394589777246842,
268
+ "learning_rate": 1.7500000000000002e-05,
269
+ "loss": 1.6437,
270
+ "step": 36
271
+ },
272
+ {
273
+ "epoch": 0.10320781032078104,
274
+ "grad_norm": 1.4186049044929165,
275
+ "learning_rate": 1.8e-05,
276
+ "loss": 1.5162,
277
+ "step": 37
278
+ },
279
+ {
280
+ "epoch": 0.10599721059972106,
281
+ "grad_norm": 1.4241876007622027,
282
+ "learning_rate": 1.8500000000000002e-05,
283
+ "loss": 1.2395,
284
+ "step": 38
285
+ },
286
+ {
287
+ "epoch": 0.1087866108786611,
288
+ "grad_norm": 1.3743075911034852,
289
+ "learning_rate": 1.9e-05,
290
+ "loss": 1.4631,
291
+ "step": 39
292
+ },
293
+ {
294
+ "epoch": 0.11157601115760112,
295
+ "grad_norm": 1.2863497247985725,
296
+ "learning_rate": 1.95e-05,
297
+ "loss": 1.3611,
298
+ "step": 40
299
+ },
300
+ {
301
+ "epoch": 0.11436541143654114,
302
+ "grad_norm": 1.3054075793205129,
303
+ "learning_rate": 2e-05,
304
+ "loss": 1.2375,
305
+ "step": 41
306
+ },
307
+ {
308
+ "epoch": 0.11715481171548117,
309
+ "grad_norm": 1.447249035073335,
310
+ "learning_rate": 1.9999892011980525e-05,
311
+ "loss": 1.4044,
312
+ "step": 42
313
+ },
314
+ {
315
+ "epoch": 0.1199442119944212,
316
+ "grad_norm": 1.3461748028257197,
317
+ "learning_rate": 1.9999568050254373e-05,
318
+ "loss": 1.4081,
319
+ "step": 43
320
+ },
321
+ {
322
+ "epoch": 0.12273361227336123,
323
+ "grad_norm": 1.6035229132022362,
324
+ "learning_rate": 1.999902812181835e-05,
325
+ "loss": 1.4167,
326
+ "step": 44
327
+ },
328
+ {
329
+ "epoch": 0.12552301255230125,
330
+ "grad_norm": 1.7750306011999208,
331
+ "learning_rate": 1.9998272238333606e-05,
332
+ "loss": 1.3533,
333
+ "step": 45
334
+ },
335
+ {
336
+ "epoch": 0.12552301255230125,
337
+ "eval_loss": 1.6828460693359375,
338
+ "eval_runtime": 158.4193,
339
+ "eval_samples_per_second": 2.007,
340
+ "eval_steps_per_second": 0.126,
341
+ "step": 45
342
+ },
343
+ {
344
+ "epoch": 0.12831241283124128,
345
+ "grad_norm": 1.354470787639867,
346
+ "learning_rate": 1.9997300416125426e-05,
347
+ "loss": 1.3901,
348
+ "step": 46
349
+ },
350
+ {
351
+ "epoch": 0.13110181311018132,
352
+ "grad_norm": 1.23248602443203,
353
+ "learning_rate": 1.999611267618283e-05,
354
+ "loss": 1.3485,
355
+ "step": 47
356
+ },
357
+ {
358
+ "epoch": 0.13389121338912133,
359
+ "grad_norm": 1.314827114627873,
360
+ "learning_rate": 1.9994709044158157e-05,
361
+ "loss": 1.3195,
362
+ "step": 48
363
+ },
364
+ {
365
+ "epoch": 0.13668061366806136,
366
+ "grad_norm": 1.3741001015735983,
367
+ "learning_rate": 1.99930895503665e-05,
368
+ "loss": 1.4394,
369
+ "step": 49
370
+ },
371
+ {
372
+ "epoch": 0.1394700139470014,
373
+ "grad_norm": 1.1624406309888304,
374
+ "learning_rate": 1.9991254229785043e-05,
375
+ "loss": 1.2943,
376
+ "step": 50
377
+ },
378
+ {
379
+ "epoch": 0.14225941422594143,
380
+ "grad_norm": 1.431188759221728,
381
+ "learning_rate": 1.998920312205231e-05,
382
+ "loss": 1.4614,
383
+ "step": 51
384
+ },
385
+ {
386
+ "epoch": 0.14504881450488144,
387
+ "grad_norm": 1.2513371987994648,
388
+ "learning_rate": 1.9986936271467316e-05,
389
+ "loss": 1.328,
390
+ "step": 52
391
+ },
392
+ {
393
+ "epoch": 0.14783821478382148,
394
+ "grad_norm": 1.6084481811469094,
395
+ "learning_rate": 1.99844537269886e-05,
396
+ "loss": 1.4579,
397
+ "step": 53
398
+ },
399
+ {
400
+ "epoch": 0.1506276150627615,
401
+ "grad_norm": 1.3643143080926854,
402
+ "learning_rate": 1.9981755542233175e-05,
403
+ "loss": 1.4471,
404
+ "step": 54
405
+ },
406
+ {
407
+ "epoch": 0.15341701534170155,
408
+ "grad_norm": 1.443453372098446,
409
+ "learning_rate": 1.9978841775475368e-05,
410
+ "loss": 1.3931,
411
+ "step": 55
412
+ },
413
+ {
414
+ "epoch": 0.15620641562064155,
415
+ "grad_norm": 1.186174044031903,
416
+ "learning_rate": 1.997571248964556e-05,
417
+ "loss": 1.2935,
418
+ "step": 56
419
+ },
420
+ {
421
+ "epoch": 0.1589958158995816,
422
+ "grad_norm": 1.3967845728055575,
423
+ "learning_rate": 1.9972367752328824e-05,
424
+ "loss": 1.3819,
425
+ "step": 57
426
+ },
427
+ {
428
+ "epoch": 0.16178521617852162,
429
+ "grad_norm": 1.585201984076838,
430
+ "learning_rate": 1.9968807635763472e-05,
431
+ "loss": 1.5391,
432
+ "step": 58
433
+ },
434
+ {
435
+ "epoch": 0.16457461645746166,
436
+ "grad_norm": 1.515327450581085,
437
+ "learning_rate": 1.9965032216839493e-05,
438
+ "loss": 1.3969,
439
+ "step": 59
440
+ },
441
+ {
442
+ "epoch": 0.16736401673640167,
443
+ "grad_norm": 1.3595894447160555,
444
+ "learning_rate": 1.996104157709689e-05,
445
+ "loss": 1.3598,
446
+ "step": 60
447
+ },
448
+ {
449
+ "epoch": 0.1701534170153417,
450
+ "grad_norm": 1.4252798961269173,
451
+ "learning_rate": 1.9956835802723916e-05,
452
+ "loss": 1.447,
453
+ "step": 61
454
+ },
455
+ {
456
+ "epoch": 0.17294281729428174,
457
+ "grad_norm": 1.4269613254937963,
458
+ "learning_rate": 1.9952414984555225e-05,
459
+ "loss": 1.4354,
460
+ "step": 62
461
+ },
462
+ {
463
+ "epoch": 0.17573221757322174,
464
+ "grad_norm": 1.5984564978735487,
465
+ "learning_rate": 1.994777921806989e-05,
466
+ "loss": 1.5493,
467
+ "step": 63
468
+ },
469
+ {
470
+ "epoch": 0.17852161785216178,
471
+ "grad_norm": 1.5396475514798837,
472
+ "learning_rate": 1.9942928603389366e-05,
473
+ "loss": 1.4095,
474
+ "step": 64
475
+ },
476
+ {
477
+ "epoch": 0.18131101813110181,
478
+ "grad_norm": 1.381818615640278,
479
+ "learning_rate": 1.9937863245275303e-05,
480
+ "loss": 1.381,
481
+ "step": 65
482
+ },
483
+ {
484
+ "epoch": 0.18410041841004185,
485
+ "grad_norm": 1.3411933307262995,
486
+ "learning_rate": 1.9932583253127302e-05,
487
+ "loss": 1.4514,
488
+ "step": 66
489
+ },
490
+ {
491
+ "epoch": 0.18688981868898186,
492
+ "grad_norm": 1.4666636741675445,
493
+ "learning_rate": 1.992708874098054e-05,
494
+ "loss": 1.378,
495
+ "step": 67
496
+ },
497
+ {
498
+ "epoch": 0.1896792189679219,
499
+ "grad_norm": 1.321297415086869,
500
+ "learning_rate": 1.9921379827503316e-05,
501
+ "loss": 1.3916,
502
+ "step": 68
503
+ },
504
+ {
505
+ "epoch": 0.19246861924686193,
506
+ "grad_norm": 1.3031514888132292,
507
+ "learning_rate": 1.991545663599448e-05,
508
+ "loss": 1.3568,
509
+ "step": 69
510
+ },
511
+ {
512
+ "epoch": 0.19525801952580196,
513
+ "grad_norm": 1.3252501098302665,
514
+ "learning_rate": 1.990931929438078e-05,
515
+ "loss": 1.3691,
516
+ "step": 70
517
+ },
518
+ {
519
+ "epoch": 0.19804741980474197,
520
+ "grad_norm": 1.3190300203004819,
521
+ "learning_rate": 1.990296793521408e-05,
522
+ "loss": 1.425,
523
+ "step": 71
524
+ },
525
+ {
526
+ "epoch": 0.200836820083682,
527
+ "grad_norm": 1.221728638105595,
528
+ "learning_rate": 1.989640269566853e-05,
529
+ "loss": 1.3669,
530
+ "step": 72
531
+ },
532
+ {
533
+ "epoch": 0.20362622036262204,
534
+ "grad_norm": 1.2519446160682244,
535
+ "learning_rate": 1.9889623717537564e-05,
536
+ "loss": 1.5065,
537
+ "step": 73
538
+ },
539
+ {
540
+ "epoch": 0.20641562064156208,
541
+ "grad_norm": 1.3192555240060455,
542
+ "learning_rate": 1.9882631147230874e-05,
543
+ "loss": 1.3077,
544
+ "step": 74
545
+ },
546
+ {
547
+ "epoch": 0.20920502092050208,
548
+ "grad_norm": 1.811281511932427,
549
+ "learning_rate": 1.987542513577122e-05,
550
+ "loss": 1.4737,
551
+ "step": 75
552
+ },
553
+ {
554
+ "epoch": 0.21199442119944212,
555
+ "grad_norm": 1.2107466822843787,
556
+ "learning_rate": 1.9868005838791185e-05,
557
+ "loss": 1.3632,
558
+ "step": 76
559
+ },
560
+ {
561
+ "epoch": 0.21478382147838215,
562
+ "grad_norm": 1.4506886695952612,
563
+ "learning_rate": 1.9860373416529804e-05,
564
+ "loss": 1.3707,
565
+ "step": 77
566
+ },
567
+ {
568
+ "epoch": 0.2175732217573222,
569
+ "grad_norm": 1.129475911714987,
570
+ "learning_rate": 1.985252803382911e-05,
571
+ "loss": 1.2879,
572
+ "step": 78
573
+ },
574
+ {
575
+ "epoch": 0.2203626220362622,
576
+ "grad_norm": 1.2378410441125478,
577
+ "learning_rate": 1.984446986013057e-05,
578
+ "loss": 1.4352,
579
+ "step": 79
580
+ },
581
+ {
582
+ "epoch": 0.22315202231520223,
583
+ "grad_norm": 1.2497274548196218,
584
+ "learning_rate": 1.983619906947144e-05,
585
+ "loss": 1.2929,
586
+ "step": 80
587
+ },
588
+ {
589
+ "epoch": 0.22594142259414227,
590
+ "grad_norm": 1.2860272115066647,
591
+ "learning_rate": 1.9827715840480962e-05,
592
+ "loss": 1.2595,
593
+ "step": 81
594
+ },
595
+ {
596
+ "epoch": 0.22873082287308227,
597
+ "grad_norm": 1.3259601590310435,
598
+ "learning_rate": 1.9819020356376562e-05,
599
+ "loss": 1.5421,
600
+ "step": 82
601
+ },
602
+ {
603
+ "epoch": 0.2315202231520223,
604
+ "grad_norm": 1.3823289295025785,
605
+ "learning_rate": 1.9810112804959867e-05,
606
+ "loss": 1.4379,
607
+ "step": 83
608
+ },
609
+ {
610
+ "epoch": 0.23430962343096234,
611
+ "grad_norm": 1.2790636696723037,
612
+ "learning_rate": 1.980099337861264e-05,
613
+ "loss": 1.4045,
614
+ "step": 84
615
+ },
616
+ {
617
+ "epoch": 0.23709902370990238,
618
+ "grad_norm": 1.2621634213682735,
619
+ "learning_rate": 1.9791662274292638e-05,
620
+ "loss": 1.3433,
621
+ "step": 85
622
+ },
623
+ {
624
+ "epoch": 0.2398884239888424,
625
+ "grad_norm": 1.4041226411604562,
626
+ "learning_rate": 1.9782119693529358e-05,
627
+ "loss": 1.6292,
628
+ "step": 86
629
+ },
630
+ {
631
+ "epoch": 0.24267782426778242,
632
+ "grad_norm": 1.4238393471976662,
633
+ "learning_rate": 1.977236584241968e-05,
634
+ "loss": 1.398,
635
+ "step": 87
636
+ },
637
+ {
638
+ "epoch": 0.24546722454672246,
639
+ "grad_norm": 1.3721757997449184,
640
+ "learning_rate": 1.9762400931623413e-05,
641
+ "loss": 1.372,
642
+ "step": 88
643
+ },
644
+ {
645
+ "epoch": 0.2482566248256625,
646
+ "grad_norm": 1.3049726438204359,
647
+ "learning_rate": 1.9752225176358757e-05,
648
+ "loss": 1.4366,
649
+ "step": 89
650
+ },
651
+ {
652
+ "epoch": 0.2510460251046025,
653
+ "grad_norm": 1.319316702022308,
654
+ "learning_rate": 1.9741838796397638e-05,
655
+ "loss": 1.2807,
656
+ "step": 90
657
+ },
658
+ {
659
+ "epoch": 0.2510460251046025,
660
+ "eval_loss": 1.6544907093048096,
661
+ "eval_runtime": 157.4295,
662
+ "eval_samples_per_second": 2.02,
663
+ "eval_steps_per_second": 0.127,
664
+ "step": 90
665
+ },
666
+ {
667
+ "epoch": 0.25383542538354253,
668
+ "grad_norm": 1.3259568867589984,
669
+ "learning_rate": 1.9731242016060985e-05,
670
+ "loss": 1.3566,
671
+ "step": 91
672
+ },
673
+ {
674
+ "epoch": 0.25662482566248257,
675
+ "grad_norm": 1.3318698765554446,
676
+ "learning_rate": 1.972043506421386e-05,
677
+ "loss": 1.3304,
678
+ "step": 92
679
+ },
680
+ {
681
+ "epoch": 0.2594142259414226,
682
+ "grad_norm": 1.210762982291795,
683
+ "learning_rate": 1.9709418174260523e-05,
684
+ "loss": 1.3663,
685
+ "step": 93
686
+ },
687
+ {
688
+ "epoch": 0.26220362622036264,
689
+ "grad_norm": 1.322366639833671,
690
+ "learning_rate": 1.9698191584139402e-05,
691
+ "loss": 1.4099,
692
+ "step": 94
693
+ },
694
+ {
695
+ "epoch": 0.2649930264993027,
696
+ "grad_norm": 1.286954377653034,
697
+ "learning_rate": 1.9686755536317945e-05,
698
+ "loss": 1.3539,
699
+ "step": 95
700
+ },
701
+ {
702
+ "epoch": 0.26778242677824265,
703
+ "grad_norm": 1.1590651579123286,
704
+ "learning_rate": 1.967511027778738e-05,
705
+ "loss": 1.2147,
706
+ "step": 96
707
+ },
708
+ {
709
+ "epoch": 0.2705718270571827,
710
+ "grad_norm": 1.3100829080105303,
711
+ "learning_rate": 1.9663256060057395e-05,
712
+ "loss": 1.4093,
713
+ "step": 97
714
+ },
715
+ {
716
+ "epoch": 0.2733612273361227,
717
+ "grad_norm": 1.346265752760783,
718
+ "learning_rate": 1.965119313915068e-05,
719
+ "loss": 1.491,
720
+ "step": 98
721
+ },
722
+ {
723
+ "epoch": 0.27615062761506276,
724
+ "grad_norm": 1.378168477135006,
725
+ "learning_rate": 1.9638921775597428e-05,
726
+ "loss": 1.5049,
727
+ "step": 99
728
+ },
729
+ {
730
+ "epoch": 0.2789400278940028,
731
+ "grad_norm": 1.4182107155054933,
732
+ "learning_rate": 1.9626442234429684e-05,
733
+ "loss": 1.5013,
734
+ "step": 100
735
+ },
736
+ {
737
+ "epoch": 0.28172942817294283,
738
+ "grad_norm": 1.3165711654218712,
739
+ "learning_rate": 1.961375478517564e-05,
740
+ "loss": 1.5105,
741
+ "step": 101
742
+ },
743
+ {
744
+ "epoch": 0.28451882845188287,
745
+ "grad_norm": 1.364780296172199,
746
+ "learning_rate": 1.9600859701853796e-05,
747
+ "loss": 1.5207,
748
+ "step": 102
749
+ },
750
+ {
751
+ "epoch": 0.28730822873082285,
752
+ "grad_norm": 1.2988922679076684,
753
+ "learning_rate": 1.958775726296706e-05,
754
+ "loss": 1.2881,
755
+ "step": 103
756
+ },
757
+ {
758
+ "epoch": 0.2900976290097629,
759
+ "grad_norm": 1.3450528168801636,
760
+ "learning_rate": 1.9574447751496706e-05,
761
+ "loss": 1.2877,
762
+ "step": 104
763
+ },
764
+ {
765
+ "epoch": 0.2928870292887029,
766
+ "grad_norm": 1.4142239844479165,
767
+ "learning_rate": 1.95609314548963e-05,
768
+ "loss": 1.5604,
769
+ "step": 105
770
+ },
771
+ {
772
+ "epoch": 0.29567642956764295,
773
+ "grad_norm": 1.3452993293539794,
774
+ "learning_rate": 1.954720866508546e-05,
775
+ "loss": 1.3447,
776
+ "step": 106
777
+ },
778
+ {
779
+ "epoch": 0.298465829846583,
780
+ "grad_norm": 1.4534491830895881,
781
+ "learning_rate": 1.953327967844356e-05,
782
+ "loss": 1.3643,
783
+ "step": 107
784
+ },
785
+ {
786
+ "epoch": 0.301255230125523,
787
+ "grad_norm": 1.350966370835706,
788
+ "learning_rate": 1.9519144795803342e-05,
789
+ "loss": 1.5196,
790
+ "step": 108
791
+ },
792
+ {
793
+ "epoch": 0.30404463040446306,
794
+ "grad_norm": 1.3487778511447879,
795
+ "learning_rate": 1.95048043224444e-05,
796
+ "loss": 1.4243,
797
+ "step": 109
798
+ },
799
+ {
800
+ "epoch": 0.3068340306834031,
801
+ "grad_norm": 1.3479508491141639,
802
+ "learning_rate": 1.94902585680866e-05,
803
+ "loss": 1.2869,
804
+ "step": 110
805
+ },
806
+ {
807
+ "epoch": 0.30962343096234307,
808
+ "grad_norm": 1.5555627133101146,
809
+ "learning_rate": 1.9475507846883377e-05,
810
+ "loss": 1.4626,
811
+ "step": 111
812
+ },
813
+ {
814
+ "epoch": 0.3124128312412831,
815
+ "grad_norm": 1.264890715784298,
816
+ "learning_rate": 1.9460552477414972e-05,
817
+ "loss": 1.2826,
818
+ "step": 112
819
+ },
820
+ {
821
+ "epoch": 0.31520223152022314,
822
+ "grad_norm": 1.2628748380778831,
823
+ "learning_rate": 1.9445392782681523e-05,
824
+ "loss": 1.232,
825
+ "step": 113
826
+ },
827
+ {
828
+ "epoch": 0.3179916317991632,
829
+ "grad_norm": 1.3550404859387195,
830
+ "learning_rate": 1.9430029090096118e-05,
831
+ "loss": 1.3034,
832
+ "step": 114
833
+ },
834
+ {
835
+ "epoch": 0.3207810320781032,
836
+ "grad_norm": 1.1496127251687267,
837
+ "learning_rate": 1.94144617314777e-05,
838
+ "loss": 1.3215,
839
+ "step": 115
840
+ },
841
+ {
842
+ "epoch": 0.32357043235704325,
843
+ "grad_norm": 1.3116085375581363,
844
+ "learning_rate": 1.939869104304392e-05,
845
+ "loss": 1.4154,
846
+ "step": 116
847
+ },
848
+ {
849
+ "epoch": 0.3263598326359833,
850
+ "grad_norm": 1.2589366985022925,
851
+ "learning_rate": 1.9382717365403854e-05,
852
+ "loss": 1.517,
853
+ "step": 117
854
+ },
855
+ {
856
+ "epoch": 0.3291492329149233,
857
+ "grad_norm": 1.3051575397158521,
858
+ "learning_rate": 1.9366541043550667e-05,
859
+ "loss": 1.3795,
860
+ "step": 118
861
+ },
862
+ {
863
+ "epoch": 0.3319386331938633,
864
+ "grad_norm": 1.514184766493585,
865
+ "learning_rate": 1.9350162426854152e-05,
866
+ "loss": 1.5974,
867
+ "step": 119
868
+ },
869
+ {
870
+ "epoch": 0.33472803347280333,
871
+ "grad_norm": 1.3773144705212048,
872
+ "learning_rate": 1.933358186905318e-05,
873
+ "loss": 1.3696,
874
+ "step": 120
875
+ },
876
+ {
877
+ "epoch": 0.33751743375174337,
878
+ "grad_norm": 1.070700275666401,
879
+ "learning_rate": 1.9316799728248074e-05,
880
+ "loss": 1.1661,
881
+ "step": 121
882
+ },
883
+ {
884
+ "epoch": 0.3403068340306834,
885
+ "grad_norm": 1.4181760004495874,
886
+ "learning_rate": 1.9299816366892865e-05,
887
+ "loss": 1.4376,
888
+ "step": 122
889
+ },
890
+ {
891
+ "epoch": 0.34309623430962344,
892
+ "grad_norm": 1.6034791315185046,
893
+ "learning_rate": 1.9282632151787462e-05,
894
+ "loss": 1.5891,
895
+ "step": 123
896
+ },
897
+ {
898
+ "epoch": 0.3458856345885635,
899
+ "grad_norm": 1.1884570183288778,
900
+ "learning_rate": 1.9265247454069736e-05,
901
+ "loss": 1.3432,
902
+ "step": 124
903
+ },
904
+ {
905
+ "epoch": 0.3486750348675035,
906
+ "grad_norm": 1.477555714731358,
907
+ "learning_rate": 1.924766264920751e-05,
908
+ "loss": 1.3683,
909
+ "step": 125
910
+ },
911
+ {
912
+ "epoch": 0.3514644351464435,
913
+ "grad_norm": 1.2504857458807423,
914
+ "learning_rate": 1.922987811699042e-05,
915
+ "loss": 1.2643,
916
+ "step": 126
917
+ },
918
+ {
919
+ "epoch": 0.3542538354253835,
920
+ "grad_norm": 1.3679090790056112,
921
+ "learning_rate": 1.9211894241521757e-05,
922
+ "loss": 1.5962,
923
+ "step": 127
924
+ },
925
+ {
926
+ "epoch": 0.35704323570432356,
927
+ "grad_norm": 1.2309836816544464,
928
+ "learning_rate": 1.9193711411210138e-05,
929
+ "loss": 1.3361,
930
+ "step": 128
931
+ },
932
+ {
933
+ "epoch": 0.3598326359832636,
934
+ "grad_norm": 1.1714321663378802,
935
+ "learning_rate": 1.917533001876113e-05,
936
+ "loss": 1.3019,
937
+ "step": 129
938
+ },
939
+ {
940
+ "epoch": 0.36262203626220363,
941
+ "grad_norm": 1.2813447334304535,
942
+ "learning_rate": 1.9156750461168768e-05,
943
+ "loss": 1.4319,
944
+ "step": 130
945
+ },
946
+ {
947
+ "epoch": 0.36541143654114366,
948
+ "grad_norm": 1.427767209588848,
949
+ "learning_rate": 1.9137973139706973e-05,
950
+ "loss": 1.4712,
951
+ "step": 131
952
+ },
953
+ {
954
+ "epoch": 0.3682008368200837,
955
+ "grad_norm": 1.3426640935974619,
956
+ "learning_rate": 1.91189984599209e-05,
957
+ "loss": 1.4397,
958
+ "step": 132
959
+ },
960
+ {
961
+ "epoch": 0.37099023709902373,
962
+ "grad_norm": 1.3173830250699687,
963
+ "learning_rate": 1.9099826831618168e-05,
964
+ "loss": 1.4799,
965
+ "step": 133
966
+ },
967
+ {
968
+ "epoch": 0.3737796373779637,
969
+ "grad_norm": 1.5260053920559482,
970
+ "learning_rate": 1.908045866886001e-05,
971
+ "loss": 1.5167,
972
+ "step": 134
973
+ },
974
+ {
975
+ "epoch": 0.37656903765690375,
976
+ "grad_norm": 1.2838799058228236,
977
+ "learning_rate": 1.9060894389952328e-05,
978
+ "loss": 1.3957,
979
+ "step": 135
980
+ },
981
+ {
982
+ "epoch": 0.37656903765690375,
983
+ "eval_loss": 1.629968523979187,
984
+ "eval_runtime": 155.8796,
985
+ "eval_samples_per_second": 2.04,
986
+ "eval_steps_per_second": 0.128,
987
+ "step": 135
988
+ },
989
+ {
990
+ "epoch": 0.3793584379358438,
991
+ "grad_norm": 1.2595723963522658,
992
+ "learning_rate": 1.9041134417436674e-05,
993
+ "loss": 1.3773,
994
+ "step": 136
995
+ },
996
+ {
997
+ "epoch": 0.3821478382147838,
998
+ "grad_norm": 1.380977078216302,
999
+ "learning_rate": 1.9021179178081107e-05,
1000
+ "loss": 1.5358,
1001
+ "step": 137
1002
+ },
1003
+ {
1004
+ "epoch": 0.38493723849372385,
1005
+ "grad_norm": 1.3245015694120374,
1006
+ "learning_rate": 1.9001029102870982e-05,
1007
+ "loss": 1.506,
1008
+ "step": 138
1009
+ },
1010
+ {
1011
+ "epoch": 0.3877266387726639,
1012
+ "grad_norm": 1.2602456347612,
1013
+ "learning_rate": 1.898068462699964e-05,
1014
+ "loss": 1.4246,
1015
+ "step": 139
1016
+ },
1017
+ {
1018
+ "epoch": 0.3905160390516039,
1019
+ "grad_norm": 1.3989105118625398,
1020
+ "learning_rate": 1.8960146189859014e-05,
1021
+ "loss": 1.4692,
1022
+ "step": 140
1023
+ },
1024
+ {
1025
+ "epoch": 0.39330543933054396,
1026
+ "grad_norm": 1.2616111800783727,
1027
+ "learning_rate": 1.8939414235030137e-05,
1028
+ "loss": 1.3755,
1029
+ "step": 141
1030
+ },
1031
+ {
1032
+ "epoch": 0.39609483960948394,
1033
+ "grad_norm": 1.2686523296470786,
1034
+ "learning_rate": 1.891848921027355e-05,
1035
+ "loss": 1.3442,
1036
+ "step": 142
1037
+ },
1038
+ {
1039
+ "epoch": 0.398884239888424,
1040
+ "grad_norm": 1.37417212405681,
1041
+ "learning_rate": 1.889737156751965e-05,
1042
+ "loss": 1.3651,
1043
+ "step": 143
1044
+ },
1045
+ {
1046
+ "epoch": 0.401673640167364,
1047
+ "grad_norm": 1.373244156828418,
1048
+ "learning_rate": 1.887606176285893e-05,
1049
+ "loss": 1.4961,
1050
+ "step": 144
1051
+ },
1052
+ {
1053
+ "epoch": 0.40446304044630405,
1054
+ "grad_norm": 1.1872961109156253,
1055
+ "learning_rate": 1.8854560256532098e-05,
1056
+ "loss": 1.2587,
1057
+ "step": 145
1058
+ },
1059
+ {
1060
+ "epoch": 0.4072524407252441,
1061
+ "grad_norm": 1.313036403643509,
1062
+ "learning_rate": 1.883286751292018e-05,
1063
+ "loss": 1.4575,
1064
+ "step": 146
1065
+ },
1066
+ {
1067
+ "epoch": 0.4100418410041841,
1068
+ "grad_norm": 1.390974479025736,
1069
+ "learning_rate": 1.8810984000534457e-05,
1070
+ "loss": 1.3891,
1071
+ "step": 147
1072
+ },
1073
+ {
1074
+ "epoch": 0.41283124128312415,
1075
+ "grad_norm": 1.3368020257438076,
1076
+ "learning_rate": 1.8788910192006363e-05,
1077
+ "loss": 1.3415,
1078
+ "step": 148
1079
+ },
1080
+ {
1081
+ "epoch": 0.41562064156206413,
1082
+ "grad_norm": 1.2329515511987332,
1083
+ "learning_rate": 1.8766646564077265e-05,
1084
+ "loss": 1.3339,
1085
+ "step": 149
1086
+ },
1087
+ {
1088
+ "epoch": 0.41841004184100417,
1089
+ "grad_norm": 1.276208090352136,
1090
+ "learning_rate": 1.8744193597588185e-05,
1091
+ "loss": 1.4495,
1092
+ "step": 150
1093
+ },
1094
+ {
1095
+ "epoch": 0.4211994421199442,
1096
+ "grad_norm": 1.1548092334953926,
1097
+ "learning_rate": 1.8721551777469397e-05,
1098
+ "loss": 1.2099,
1099
+ "step": 151
1100
+ },
1101
+ {
1102
+ "epoch": 0.42398884239888424,
1103
+ "grad_norm": 1.3184986062317001,
1104
+ "learning_rate": 1.869872159272997e-05,
1105
+ "loss": 1.2136,
1106
+ "step": 152
1107
+ },
1108
+ {
1109
+ "epoch": 0.42677824267782427,
1110
+ "grad_norm": 1.1962726760049636,
1111
+ "learning_rate": 1.8675703536447178e-05,
1112
+ "loss": 1.2598,
1113
+ "step": 153
1114
+ },
1115
+ {
1116
+ "epoch": 0.4295676429567643,
1117
+ "grad_norm": 1.3302312487017707,
1118
+ "learning_rate": 1.8652498105755898e-05,
1119
+ "loss": 1.3719,
1120
+ "step": 154
1121
+ },
1122
+ {
1123
+ "epoch": 0.43235704323570434,
1124
+ "grad_norm": 1.55872330252289,
1125
+ "learning_rate": 1.862910580183782e-05,
1126
+ "loss": 1.4867,
1127
+ "step": 155
1128
+ },
1129
+ {
1130
+ "epoch": 0.4351464435146444,
1131
+ "grad_norm": 1.3004017172912832,
1132
+ "learning_rate": 1.8605527129910663e-05,
1133
+ "loss": 1.3089,
1134
+ "step": 156
1135
+ },
1136
+ {
1137
+ "epoch": 0.43793584379358436,
1138
+ "grad_norm": 1.2650294480346138,
1139
+ "learning_rate": 1.858176259921724e-05,
1140
+ "loss": 1.2856,
1141
+ "step": 157
1142
+ },
1143
+ {
1144
+ "epoch": 0.4407252440725244,
1145
+ "grad_norm": 1.414189467122214,
1146
+ "learning_rate": 1.8557812723014476e-05,
1147
+ "loss": 1.4827,
1148
+ "step": 158
1149
+ },
1150
+ {
1151
+ "epoch": 0.4435146443514644,
1152
+ "grad_norm": 1.2634959665686347,
1153
+ "learning_rate": 1.853367801856231e-05,
1154
+ "loss": 1.3568,
1155
+ "step": 159
1156
+ },
1157
+ {
1158
+ "epoch": 0.44630404463040446,
1159
+ "grad_norm": 1.3164047095667806,
1160
+ "learning_rate": 1.8509359007112523e-05,
1161
+ "loss": 1.5306,
1162
+ "step": 160
1163
+ },
1164
+ {
1165
+ "epoch": 0.4490934449093445,
1166
+ "grad_norm": 1.3232600329575266,
1167
+ "learning_rate": 1.8484856213897496e-05,
1168
+ "loss": 1.4279,
1169
+ "step": 161
1170
+ },
1171
+ {
1172
+ "epoch": 0.45188284518828453,
1173
+ "grad_norm": 1.286194256752095,
1174
+ "learning_rate": 1.8460170168118857e-05,
1175
+ "loss": 1.2656,
1176
+ "step": 162
1177
+ },
1178
+ {
1179
+ "epoch": 0.45467224546722457,
1180
+ "grad_norm": 1.393553888811773,
1181
+ "learning_rate": 1.843530140293603e-05,
1182
+ "loss": 1.4003,
1183
+ "step": 163
1184
+ },
1185
+ {
1186
+ "epoch": 0.45746164574616455,
1187
+ "grad_norm": 1.2952418440030533,
1188
+ "learning_rate": 1.841025045545477e-05,
1189
+ "loss": 1.365,
1190
+ "step": 164
1191
+ },
1192
+ {
1193
+ "epoch": 0.4602510460251046,
1194
+ "grad_norm": 1.2556167978994803,
1195
+ "learning_rate": 1.8385017866715507e-05,
1196
+ "loss": 1.3313,
1197
+ "step": 165
1198
+ },
1199
+ {
1200
+ "epoch": 0.4630404463040446,
1201
+ "grad_norm": 1.3067564204278406,
1202
+ "learning_rate": 1.8359604181681703e-05,
1203
+ "loss": 1.4114,
1204
+ "step": 166
1205
+ },
1206
+ {
1207
+ "epoch": 0.46582984658298465,
1208
+ "grad_norm": 1.2088590174928828,
1209
+ "learning_rate": 1.833400994922806e-05,
1210
+ "loss": 1.1866,
1211
+ "step": 167
1212
+ },
1213
+ {
1214
+ "epoch": 0.4686192468619247,
1215
+ "grad_norm": 1.334753865529686,
1216
+ "learning_rate": 1.8308235722128674e-05,
1217
+ "loss": 1.2744,
1218
+ "step": 168
1219
+ },
1220
+ {
1221
+ "epoch": 0.4714086471408647,
1222
+ "grad_norm": 1.3612556693462265,
1223
+ "learning_rate": 1.8282282057045087e-05,
1224
+ "loss": 1.3558,
1225
+ "step": 169
1226
+ },
1227
+ {
1228
+ "epoch": 0.47419804741980476,
1229
+ "grad_norm": 1.2660924285869366,
1230
+ "learning_rate": 1.8256149514514284e-05,
1231
+ "loss": 1.2804,
1232
+ "step": 170
1233
+ },
1234
+ {
1235
+ "epoch": 0.4769874476987448,
1236
+ "grad_norm": 1.3192403133250346,
1237
+ "learning_rate": 1.8229838658936566e-05,
1238
+ "loss": 1.3874,
1239
+ "step": 171
1240
+ },
1241
+ {
1242
+ "epoch": 0.4797768479776848,
1243
+ "grad_norm": 1.3924722985838183,
1244
+ "learning_rate": 1.8203350058563366e-05,
1245
+ "loss": 1.5013,
1246
+ "step": 172
1247
+ },
1248
+ {
1249
+ "epoch": 0.4825662482566248,
1250
+ "grad_norm": 1.2874077296032287,
1251
+ "learning_rate": 1.8176684285484985e-05,
1252
+ "loss": 1.3261,
1253
+ "step": 173
1254
+ },
1255
+ {
1256
+ "epoch": 0.48535564853556484,
1257
+ "grad_norm": 1.309597924994118,
1258
+ "learning_rate": 1.814984191561823e-05,
1259
+ "loss": 1.4157,
1260
+ "step": 174
1261
+ },
1262
+ {
1263
+ "epoch": 0.4881450488145049,
1264
+ "grad_norm": 1.1912282212302885,
1265
+ "learning_rate": 1.8122823528693966e-05,
1266
+ "loss": 1.2585,
1267
+ "step": 175
1268
+ },
1269
+ {
1270
+ "epoch": 0.4909344490934449,
1271
+ "grad_norm": 1.2765007106705224,
1272
+ "learning_rate": 1.809562970824462e-05,
1273
+ "loss": 1.3437,
1274
+ "step": 176
1275
+ },
1276
+ {
1277
+ "epoch": 0.49372384937238495,
1278
+ "grad_norm": 1.2571675750543854,
1279
+ "learning_rate": 1.8068261041591548e-05,
1280
+ "loss": 1.4383,
1281
+ "step": 177
1282
+ },
1283
+ {
1284
+ "epoch": 0.496513249651325,
1285
+ "grad_norm": 1.1735759325774886,
1286
+ "learning_rate": 1.8040718119832378e-05,
1287
+ "loss": 1.3234,
1288
+ "step": 178
1289
+ },
1290
+ {
1291
+ "epoch": 0.499302649930265,
1292
+ "grad_norm": 1.1634243266804525,
1293
+ "learning_rate": 1.8013001537828213e-05,
1294
+ "loss": 1.2169,
1295
+ "step": 179
1296
+ },
1297
+ {
1298
+ "epoch": 0.502092050209205,
1299
+ "grad_norm": 1.2251535095230563,
1300
+ "learning_rate": 1.7985111894190827e-05,
1301
+ "loss": 1.2727,
1302
+ "step": 180
1303
+ },
1304
+ {
1305
+ "epoch": 0.502092050209205,
1306
+ "eval_loss": 1.6175622940063477,
1307
+ "eval_runtime": 183.4681,
1308
+ "eval_samples_per_second": 1.733,
1309
+ "eval_steps_per_second": 0.109,
1310
+ "step": 180
1311
+ },
1312
+ {
1313
+ "epoch": 0.504881450488145,
1314
+ "grad_norm": 1.2927502437198448,
1315
+ "learning_rate": 1.7957049791269684e-05,
1316
+ "loss": 1.3752,
1317
+ "step": 181
1318
+ },
1319
+ {
1320
+ "epoch": 0.5076708507670851,
1321
+ "grad_norm": 1.2697109314581512,
1322
+ "learning_rate": 1.7928815835138976e-05,
1323
+ "loss": 1.502,
1324
+ "step": 182
1325
+ },
1326
+ {
1327
+ "epoch": 0.5104602510460251,
1328
+ "grad_norm": 1.3004080082594036,
1329
+ "learning_rate": 1.79004106355845e-05,
1330
+ "loss": 1.49,
1331
+ "step": 183
1332
+ },
1333
+ {
1334
+ "epoch": 0.5132496513249651,
1335
+ "grad_norm": 1.2620161230293547,
1336
+ "learning_rate": 1.7871834806090502e-05,
1337
+ "loss": 1.258,
1338
+ "step": 184
1339
+ },
1340
+ {
1341
+ "epoch": 0.5160390516039052,
1342
+ "grad_norm": 1.2238216229925647,
1343
+ "learning_rate": 1.7843088963826437e-05,
1344
+ "loss": 1.2829,
1345
+ "step": 185
1346
+ },
1347
+ {
1348
+ "epoch": 0.5188284518828452,
1349
+ "grad_norm": 1.2320397894400665,
1350
+ "learning_rate": 1.7814173729633607e-05,
1351
+ "loss": 1.2767,
1352
+ "step": 186
1353
+ },
1354
+ {
1355
+ "epoch": 0.5216178521617852,
1356
+ "grad_norm": 1.3637977851134857,
1357
+ "learning_rate": 1.7785089728011798e-05,
1358
+ "loss": 1.3635,
1359
+ "step": 187
1360
+ },
1361
+ {
1362
+ "epoch": 0.5244072524407253,
1363
+ "grad_norm": 1.2685139570979558,
1364
+ "learning_rate": 1.775583758710575e-05,
1365
+ "loss": 1.4145,
1366
+ "step": 188
1367
+ },
1368
+ {
1369
+ "epoch": 0.5271966527196653,
1370
+ "grad_norm": 1.3355216229358506,
1371
+ "learning_rate": 1.772641793869162e-05,
1372
+ "loss": 1.3068,
1373
+ "step": 189
1374
+ },
1375
+ {
1376
+ "epoch": 0.5299860529986054,
1377
+ "grad_norm": 1.3597618241983527,
1378
+ "learning_rate": 1.7696831418163318e-05,
1379
+ "loss": 1.4446,
1380
+ "step": 190
1381
+ },
1382
+ {
1383
+ "epoch": 0.5327754532775453,
1384
+ "grad_norm": 1.2542157271291101,
1385
+ "learning_rate": 1.7667078664518796e-05,
1386
+ "loss": 1.4184,
1387
+ "step": 191
1388
+ },
1389
+ {
1390
+ "epoch": 0.5355648535564853,
1391
+ "grad_norm": 1.3244864911198961,
1392
+ "learning_rate": 1.763716032034624e-05,
1393
+ "loss": 1.3889,
1394
+ "step": 192
1395
+ },
1396
+ {
1397
+ "epoch": 0.5383542538354253,
1398
+ "grad_norm": 1.261772883234437,
1399
+ "learning_rate": 1.7607077031810204e-05,
1400
+ "loss": 1.343,
1401
+ "step": 193
1402
+ },
1403
+ {
1404
+ "epoch": 0.5411436541143654,
1405
+ "grad_norm": 1.404601026428458,
1406
+ "learning_rate": 1.7576829448637628e-05,
1407
+ "loss": 1.4606,
1408
+ "step": 194
1409
+ },
1410
+ {
1411
+ "epoch": 0.5439330543933054,
1412
+ "grad_norm": 1.3091211707895143,
1413
+ "learning_rate": 1.7546418224103838e-05,
1414
+ "loss": 1.2931,
1415
+ "step": 195
1416
+ },
1417
+ {
1418
+ "epoch": 0.5467224546722455,
1419
+ "grad_norm": 1.2631117467582635,
1420
+ "learning_rate": 1.7515844015018416e-05,
1421
+ "loss": 1.2999,
1422
+ "step": 196
1423
+ },
1424
+ {
1425
+ "epoch": 0.5495118549511855,
1426
+ "grad_norm": 1.4560900819743723,
1427
+ "learning_rate": 1.7485107481711014e-05,
1428
+ "loss": 1.5475,
1429
+ "step": 197
1430
+ },
1431
+ {
1432
+ "epoch": 0.5523012552301255,
1433
+ "grad_norm": 1.4468059745173558,
1434
+ "learning_rate": 1.7454209288017105e-05,
1435
+ "loss": 1.4191,
1436
+ "step": 198
1437
+ },
1438
+ {
1439
+ "epoch": 0.5550906555090656,
1440
+ "grad_norm": 1.315266510590906,
1441
+ "learning_rate": 1.7423150101263645e-05,
1442
+ "loss": 1.4005,
1443
+ "step": 199
1444
+ },
1445
+ {
1446
+ "epoch": 0.5578800557880056,
1447
+ "grad_norm": 1.474348076413936,
1448
+ "learning_rate": 1.7391930592254636e-05,
1449
+ "loss": 1.3192,
1450
+ "step": 200
1451
+ },
1452
+ {
1453
+ "epoch": 0.5606694560669456,
1454
+ "grad_norm": 1.3652476074142217,
1455
+ "learning_rate": 1.7360551435256673e-05,
1456
+ "loss": 1.2574,
1457
+ "step": 201
1458
+ },
1459
+ {
1460
+ "epoch": 0.5634588563458857,
1461
+ "grad_norm": 1.3798373173542198,
1462
+ "learning_rate": 1.7329013307984363e-05,
1463
+ "loss": 1.3117,
1464
+ "step": 202
1465
+ },
1466
+ {
1467
+ "epoch": 0.5662482566248257,
1468
+ "grad_norm": 1.209790296025406,
1469
+ "learning_rate": 1.729731689158568e-05,
1470
+ "loss": 1.3107,
1471
+ "step": 203
1472
+ },
1473
+ {
1474
+ "epoch": 0.5690376569037657,
1475
+ "grad_norm": 1.1868593994582228,
1476
+ "learning_rate": 1.726546287062727e-05,
1477
+ "loss": 1.2752,
1478
+ "step": 204
1479
+ },
1480
+ {
1481
+ "epoch": 0.5718270571827058,
1482
+ "grad_norm": 1.3235161134829174,
1483
+ "learning_rate": 1.7233451933079663e-05,
1484
+ "loss": 1.3399,
1485
+ "step": 205
1486
+ },
1487
+ {
1488
+ "epoch": 0.5746164574616457,
1489
+ "grad_norm": 1.353859989844425,
1490
+ "learning_rate": 1.720128477030241e-05,
1491
+ "loss": 1.3181,
1492
+ "step": 206
1493
+ },
1494
+ {
1495
+ "epoch": 0.5774058577405857,
1496
+ "grad_norm": 1.3778332664676014,
1497
+ "learning_rate": 1.7168962077029146e-05,
1498
+ "loss": 1.372,
1499
+ "step": 207
1500
+ },
1501
+ {
1502
+ "epoch": 0.5801952580195258,
1503
+ "grad_norm": 1.3065213179222184,
1504
+ "learning_rate": 1.7136484551352608e-05,
1505
+ "loss": 1.2717,
1506
+ "step": 208
1507
+ },
1508
+ {
1509
+ "epoch": 0.5829846582984658,
1510
+ "grad_norm": 1.3461935265014031,
1511
+ "learning_rate": 1.7103852894709517e-05,
1512
+ "loss": 1.3716,
1513
+ "step": 209
1514
+ },
1515
+ {
1516
+ "epoch": 0.5857740585774058,
1517
+ "grad_norm": 1.5238199082532498,
1518
+ "learning_rate": 1.7071067811865477e-05,
1519
+ "loss": 1.4732,
1520
+ "step": 210
1521
+ },
1522
+ {
1523
+ "epoch": 0.5885634588563459,
1524
+ "grad_norm": 1.2661008215508678,
1525
+ "learning_rate": 1.7038130010899716e-05,
1526
+ "loss": 1.3884,
1527
+ "step": 211
1528
+ },
1529
+ {
1530
+ "epoch": 0.5913528591352859,
1531
+ "grad_norm": 1.3340050909328596,
1532
+ "learning_rate": 1.7005040203189818e-05,
1533
+ "loss": 1.3851,
1534
+ "step": 212
1535
+ },
1536
+ {
1537
+ "epoch": 0.5941422594142259,
1538
+ "grad_norm": 1.2956837583785732,
1539
+ "learning_rate": 1.6971799103396332e-05,
1540
+ "loss": 1.4221,
1541
+ "step": 213
1542
+ },
1543
+ {
1544
+ "epoch": 0.596931659693166,
1545
+ "grad_norm": 1.1561140429998333,
1546
+ "learning_rate": 1.693840742944738e-05,
1547
+ "loss": 1.2218,
1548
+ "step": 214
1549
+ },
1550
+ {
1551
+ "epoch": 0.599721059972106,
1552
+ "grad_norm": 1.313764720344347,
1553
+ "learning_rate": 1.6904865902523098e-05,
1554
+ "loss": 1.264,
1555
+ "step": 215
1556
+ },
1557
+ {
1558
+ "epoch": 0.602510460251046,
1559
+ "grad_norm": 1.3302384495066342,
1560
+ "learning_rate": 1.68711752470401e-05,
1561
+ "loss": 1.2776,
1562
+ "step": 216
1563
+ },
1564
+ {
1565
+ "epoch": 0.6052998605299861,
1566
+ "grad_norm": 1.2191909895274562,
1567
+ "learning_rate": 1.6837336190635824e-05,
1568
+ "loss": 1.3539,
1569
+ "step": 217
1570
+ },
1571
+ {
1572
+ "epoch": 0.6080892608089261,
1573
+ "grad_norm": 1.3373684634352505,
1574
+ "learning_rate": 1.6803349464152798e-05,
1575
+ "loss": 1.4503,
1576
+ "step": 218
1577
+ },
1578
+ {
1579
+ "epoch": 0.6108786610878661,
1580
+ "grad_norm": 1.363619941169918,
1581
+ "learning_rate": 1.6769215801622884e-05,
1582
+ "loss": 1.5192,
1583
+ "step": 219
1584
+ },
1585
+ {
1586
+ "epoch": 0.6136680613668062,
1587
+ "grad_norm": 1.4606054239168875,
1588
+ "learning_rate": 1.6734935940251404e-05,
1589
+ "loss": 1.5595,
1590
+ "step": 220
1591
+ },
1592
+ {
1593
+ "epoch": 0.6164574616457462,
1594
+ "grad_norm": 1.231599938466119,
1595
+ "learning_rate": 1.6700510620401223e-05,
1596
+ "loss": 1.2604,
1597
+ "step": 221
1598
+ },
1599
+ {
1600
+ "epoch": 0.6192468619246861,
1601
+ "grad_norm": 1.5101877585841448,
1602
+ "learning_rate": 1.666594058557677e-05,
1603
+ "loss": 1.4834,
1604
+ "step": 222
1605
+ },
1606
+ {
1607
+ "epoch": 0.6220362622036262,
1608
+ "grad_norm": 1.39399625488815,
1609
+ "learning_rate": 1.6631226582407954e-05,
1610
+ "loss": 1.3877,
1611
+ "step": 223
1612
+ },
1613
+ {
1614
+ "epoch": 0.6248256624825662,
1615
+ "grad_norm": 1.453409874433298,
1616
+ "learning_rate": 1.659636936063407e-05,
1617
+ "loss": 1.5218,
1618
+ "step": 224
1619
+ },
1620
+ {
1621
+ "epoch": 0.6276150627615062,
1622
+ "grad_norm": 1.2538819898580684,
1623
+ "learning_rate": 1.6561369673087588e-05,
1624
+ "loss": 1.2438,
1625
+ "step": 225
1626
+ },
1627
+ {
1628
+ "epoch": 0.6276150627615062,
1629
+ "eval_loss": 1.6073960065841675,
1630
+ "eval_runtime": 160.2272,
1631
+ "eval_samples_per_second": 1.985,
1632
+ "eval_steps_per_second": 0.125,
1633
+ "step": 225
1634
+ },
1635
+ {
1636
+ "epoch": 0.6304044630404463,
1637
+ "grad_norm": 1.5406663292943368,
1638
+ "learning_rate": 1.6526228275677892e-05,
1639
+ "loss": 1.4646,
1640
+ "step": 226
1641
+ },
1642
+ {
1643
+ "epoch": 0.6331938633193863,
1644
+ "grad_norm": 1.2629740380853072,
1645
+ "learning_rate": 1.649094592737497e-05,
1646
+ "loss": 1.3459,
1647
+ "step": 227
1648
+ },
1649
+ {
1650
+ "epoch": 0.6359832635983264,
1651
+ "grad_norm": 1.302608797313638,
1652
+ "learning_rate": 1.6455523390193e-05,
1653
+ "loss": 1.2689,
1654
+ "step": 228
1655
+ },
1656
+ {
1657
+ "epoch": 0.6387726638772664,
1658
+ "grad_norm": 1.1773849504199476,
1659
+ "learning_rate": 1.641996142917391e-05,
1660
+ "loss": 1.2342,
1661
+ "step": 229
1662
+ },
1663
+ {
1664
+ "epoch": 0.6415620641562064,
1665
+ "grad_norm": 1.3493891917660983,
1666
+ "learning_rate": 1.638426081237085e-05,
1667
+ "loss": 1.2569,
1668
+ "step": 230
1669
+ },
1670
+ {
1671
+ "epoch": 0.6443514644351465,
1672
+ "grad_norm": 1.388250997836812,
1673
+ "learning_rate": 1.63484223108316e-05,
1674
+ "loss": 1.3979,
1675
+ "step": 231
1676
+ },
1677
+ {
1678
+ "epoch": 0.6471408647140865,
1679
+ "grad_norm": 1.1928671464502947,
1680
+ "learning_rate": 1.6312446698581915e-05,
1681
+ "loss": 1.23,
1682
+ "step": 232
1683
+ },
1684
+ {
1685
+ "epoch": 0.6499302649930265,
1686
+ "grad_norm": 1.2633309780727078,
1687
+ "learning_rate": 1.6276334752608823e-05,
1688
+ "loss": 1.4891,
1689
+ "step": 233
1690
+ },
1691
+ {
1692
+ "epoch": 0.6527196652719666,
1693
+ "grad_norm": 1.304711639719216,
1694
+ "learning_rate": 1.624008725284383e-05,
1695
+ "loss": 1.3034,
1696
+ "step": 234
1697
+ },
1698
+ {
1699
+ "epoch": 0.6555090655509066,
1700
+ "grad_norm": 1.2217261096543481,
1701
+ "learning_rate": 1.6203704982146073e-05,
1702
+ "loss": 1.2068,
1703
+ "step": 235
1704
+ },
1705
+ {
1706
+ "epoch": 0.6582984658298466,
1707
+ "grad_norm": 1.288441898801668,
1708
+ "learning_rate": 1.6167188726285433e-05,
1709
+ "loss": 1.2461,
1710
+ "step": 236
1711
+ },
1712
+ {
1713
+ "epoch": 0.6610878661087866,
1714
+ "grad_norm": 1.590492811559281,
1715
+ "learning_rate": 1.613053927392553e-05,
1716
+ "loss": 1.5137,
1717
+ "step": 237
1718
+ },
1719
+ {
1720
+ "epoch": 0.6638772663877266,
1721
+ "grad_norm": 1.2477326342958976,
1722
+ "learning_rate": 1.6093757416606725e-05,
1723
+ "loss": 1.3189,
1724
+ "step": 238
1725
+ },
1726
+ {
1727
+ "epoch": 0.6666666666666666,
1728
+ "grad_norm": 1.2127950915475818,
1729
+ "learning_rate": 1.6056843948729e-05,
1730
+ "loss": 1.2682,
1731
+ "step": 239
1732
+ },
1733
+ {
1734
+ "epoch": 0.6694560669456067,
1735
+ "grad_norm": 1.4271242774536947,
1736
+ "learning_rate": 1.6019799667534813e-05,
1737
+ "loss": 1.5004,
1738
+ "step": 240
1739
+ },
1740
+ {
1741
+ "epoch": 0.6722454672245467,
1742
+ "grad_norm": 1.3666477545942404,
1743
+ "learning_rate": 1.5982625373091877e-05,
1744
+ "loss": 1.5493,
1745
+ "step": 241
1746
+ },
1747
+ {
1748
+ "epoch": 0.6750348675034867,
1749
+ "grad_norm": 1.1483091382024733,
1750
+ "learning_rate": 1.594532186827588e-05,
1751
+ "loss": 1.3191,
1752
+ "step": 242
1753
+ },
1754
+ {
1755
+ "epoch": 0.6778242677824268,
1756
+ "grad_norm": 3.1947518624580136,
1757
+ "learning_rate": 1.5907889958753134e-05,
1758
+ "loss": 1.3046,
1759
+ "step": 243
1760
+ },
1761
+ {
1762
+ "epoch": 0.6806136680613668,
1763
+ "grad_norm": 1.3099413581809563,
1764
+ "learning_rate": 1.5870330452963206e-05,
1765
+ "loss": 1.2706,
1766
+ "step": 244
1767
+ },
1768
+ {
1769
+ "epoch": 0.6834030683403068,
1770
+ "grad_norm": 1.461185290486486,
1771
+ "learning_rate": 1.5832644162101417e-05,
1772
+ "loss": 1.4886,
1773
+ "step": 245
1774
+ },
1775
+ {
1776
+ "epoch": 0.6861924686192469,
1777
+ "grad_norm": 1.2442836680144551,
1778
+ "learning_rate": 1.5794831900101352e-05,
1779
+ "loss": 1.27,
1780
+ "step": 246
1781
+ },
1782
+ {
1783
+ "epoch": 0.6889818688981869,
1784
+ "grad_norm": 1.248119979843789,
1785
+ "learning_rate": 1.5756894483617268e-05,
1786
+ "loss": 1.2831,
1787
+ "step": 247
1788
+ },
1789
+ {
1790
+ "epoch": 0.691771269177127,
1791
+ "grad_norm": 1.3703323165735684,
1792
+ "learning_rate": 1.571883273200646e-05,
1793
+ "loss": 1.5584,
1794
+ "step": 248
1795
+ },
1796
+ {
1797
+ "epoch": 0.694560669456067,
1798
+ "grad_norm": 1.3183958125038406,
1799
+ "learning_rate": 1.568064746731156e-05,
1800
+ "loss": 1.5037,
1801
+ "step": 249
1802
+ },
1803
+ {
1804
+ "epoch": 0.697350069735007,
1805
+ "grad_norm": 1.1844817674941481,
1806
+ "learning_rate": 1.564233951424279e-05,
1807
+ "loss": 1.4209,
1808
+ "step": 250
1809
+ },
1810
+ {
1811
+ "epoch": 0.700139470013947,
1812
+ "grad_norm": 1.5190227626319008,
1813
+ "learning_rate": 1.560390970016015e-05,
1814
+ "loss": 1.4738,
1815
+ "step": 251
1816
+ },
1817
+ {
1818
+ "epoch": 0.702928870292887,
1819
+ "grad_norm": 1.2922540957047237,
1820
+ "learning_rate": 1.5565358855055535e-05,
1821
+ "loss": 1.4749,
1822
+ "step": 252
1823
+ },
1824
+ {
1825
+ "epoch": 0.705718270571827,
1826
+ "grad_norm": 1.2755003379456324,
1827
+ "learning_rate": 1.552668781153484e-05,
1828
+ "loss": 1.3514,
1829
+ "step": 253
1830
+ },
1831
+ {
1832
+ "epoch": 0.708507670850767,
1833
+ "grad_norm": 1.333525201907311,
1834
+ "learning_rate": 1.5487897404799932e-05,
1835
+ "loss": 1.4833,
1836
+ "step": 254
1837
+ },
1838
+ {
1839
+ "epoch": 0.7112970711297071,
1840
+ "grad_norm": 1.2366281388101243,
1841
+ "learning_rate": 1.5448988472630654e-05,
1842
+ "loss": 1.3055,
1843
+ "step": 255
1844
+ },
1845
+ {
1846
+ "epoch": 0.7140864714086471,
1847
+ "grad_norm": 1.2656677555514606,
1848
+ "learning_rate": 1.5409961855366718e-05,
1849
+ "loss": 1.2858,
1850
+ "step": 256
1851
+ },
1852
+ {
1853
+ "epoch": 0.7168758716875872,
1854
+ "grad_norm": 1.3893320030810534,
1855
+ "learning_rate": 1.5370818395889536e-05,
1856
+ "loss": 1.2958,
1857
+ "step": 257
1858
+ },
1859
+ {
1860
+ "epoch": 0.7196652719665272,
1861
+ "grad_norm": 1.2262571768039963,
1862
+ "learning_rate": 1.533155893960405e-05,
1863
+ "loss": 1.3797,
1864
+ "step": 258
1865
+ },
1866
+ {
1867
+ "epoch": 0.7224546722454672,
1868
+ "grad_norm": 1.3752885859305817,
1869
+ "learning_rate": 1.5292184334420434e-05,
1870
+ "loss": 1.5324,
1871
+ "step": 259
1872
+ },
1873
+ {
1874
+ "epoch": 0.7252440725244073,
1875
+ "grad_norm": 1.349973458041763,
1876
+ "learning_rate": 1.5252695430735824e-05,
1877
+ "loss": 1.3828,
1878
+ "step": 260
1879
+ },
1880
+ {
1881
+ "epoch": 0.7280334728033473,
1882
+ "grad_norm": 1.214154217005644,
1883
+ "learning_rate": 1.521309308141592e-05,
1884
+ "loss": 1.3081,
1885
+ "step": 261
1886
+ },
1887
+ {
1888
+ "epoch": 0.7308228730822873,
1889
+ "grad_norm": 1.2681774984772773,
1890
+ "learning_rate": 1.5173378141776569e-05,
1891
+ "loss": 1.5341,
1892
+ "step": 262
1893
+ },
1894
+ {
1895
+ "epoch": 0.7336122733612274,
1896
+ "grad_norm": 1.2265782510592125,
1897
+ "learning_rate": 1.5133551469565313e-05,
1898
+ "loss": 1.2877,
1899
+ "step": 263
1900
+ },
1901
+ {
1902
+ "epoch": 0.7364016736401674,
1903
+ "grad_norm": 1.2084693848644321,
1904
+ "learning_rate": 1.5093613924942837e-05,
1905
+ "loss": 1.3773,
1906
+ "step": 264
1907
+ },
1908
+ {
1909
+ "epoch": 0.7391910739191074,
1910
+ "grad_norm": 1.1437825395668497,
1911
+ "learning_rate": 1.5053566370464416e-05,
1912
+ "loss": 1.2808,
1913
+ "step": 265
1914
+ },
1915
+ {
1916
+ "epoch": 0.7419804741980475,
1917
+ "grad_norm": 1.3197967431830404,
1918
+ "learning_rate": 1.5013409671061267e-05,
1919
+ "loss": 1.289,
1920
+ "step": 266
1921
+ },
1922
+ {
1923
+ "epoch": 0.7447698744769874,
1924
+ "grad_norm": 1.382635793553147,
1925
+ "learning_rate": 1.4973144694021874e-05,
1926
+ "loss": 1.2638,
1927
+ "step": 267
1928
+ },
1929
+ {
1930
+ "epoch": 0.7475592747559274,
1931
+ "grad_norm": 1.3988324864350785,
1932
+ "learning_rate": 1.4932772308973267e-05,
1933
+ "loss": 1.3403,
1934
+ "step": 268
1935
+ },
1936
+ {
1937
+ "epoch": 0.7503486750348675,
1938
+ "grad_norm": 1.2731407752985473,
1939
+ "learning_rate": 1.4892293387862221e-05,
1940
+ "loss": 1.3312,
1941
+ "step": 269
1942
+ },
1943
+ {
1944
+ "epoch": 0.7531380753138075,
1945
+ "grad_norm": 1.1572040227672067,
1946
+ "learning_rate": 1.4851708804936449e-05,
1947
+ "loss": 1.3147,
1948
+ "step": 270
1949
+ },
1950
+ {
1951
+ "epoch": 0.7531380753138075,
1952
+ "eval_loss": 1.5958304405212402,
1953
+ "eval_runtime": 158.3728,
1954
+ "eval_samples_per_second": 2.008,
1955
+ "eval_steps_per_second": 0.126,
1956
+ "step": 270
1957
+ },
1958
+ {
1959
+ "epoch": 0.7559274755927475,
1960
+ "grad_norm": 1.3990082254921117,
1961
+ "learning_rate": 1.4811019436725684e-05,
1962
+ "loss": 1.3162,
1963
+ "step": 271
1964
+ },
1965
+ {
1966
+ "epoch": 0.7587168758716876,
1967
+ "grad_norm": 1.2418130090749915,
1968
+ "learning_rate": 1.4770226162022799e-05,
1969
+ "loss": 1.2594,
1970
+ "step": 272
1971
+ },
1972
+ {
1973
+ "epoch": 0.7615062761506276,
1974
+ "grad_norm": 1.1503772215035513,
1975
+ "learning_rate": 1.472932986186477e-05,
1976
+ "loss": 1.2827,
1977
+ "step": 273
1978
+ },
1979
+ {
1980
+ "epoch": 0.7642956764295676,
1981
+ "grad_norm": 1.459940130878171,
1982
+ "learning_rate": 1.4688331419513693e-05,
1983
+ "loss": 1.3126,
1984
+ "step": 274
1985
+ },
1986
+ {
1987
+ "epoch": 0.7670850767085077,
1988
+ "grad_norm": 1.3252355330703371,
1989
+ "learning_rate": 1.4647231720437687e-05,
1990
+ "loss": 1.4367,
1991
+ "step": 275
1992
+ },
1993
+ {
1994
+ "epoch": 0.7698744769874477,
1995
+ "grad_norm": 1.4888619740632927,
1996
+ "learning_rate": 1.4606031652291772e-05,
1997
+ "loss": 1.4275,
1998
+ "step": 276
1999
+ },
2000
+ {
2001
+ "epoch": 0.7726638772663877,
2002
+ "grad_norm": 1.3069850449838218,
2003
+ "learning_rate": 1.4564732104898702e-05,
2004
+ "loss": 1.3533,
2005
+ "step": 277
2006
+ },
2007
+ {
2008
+ "epoch": 0.7754532775453278,
2009
+ "grad_norm": 1.268759058345783,
2010
+ "learning_rate": 1.4523333970229741e-05,
2011
+ "loss": 1.337,
2012
+ "step": 278
2013
+ },
2014
+ {
2015
+ "epoch": 0.7782426778242678,
2016
+ "grad_norm": 1.1520224898754043,
2017
+ "learning_rate": 1.4481838142385403e-05,
2018
+ "loss": 1.2454,
2019
+ "step": 279
2020
+ },
2021
+ {
2022
+ "epoch": 0.7810320781032078,
2023
+ "grad_norm": 1.3031384394432615,
2024
+ "learning_rate": 1.4440245517576141e-05,
2025
+ "loss": 1.4125,
2026
+ "step": 280
2027
+ },
2028
+ {
2029
+ "epoch": 0.7838214783821479,
2030
+ "grad_norm": 1.4528549714436811,
2031
+ "learning_rate": 1.4398556994102996e-05,
2032
+ "loss": 1.3641,
2033
+ "step": 281
2034
+ },
2035
+ {
2036
+ "epoch": 0.7866108786610879,
2037
+ "grad_norm": 1.4236584359239257,
2038
+ "learning_rate": 1.4356773472338174e-05,
2039
+ "loss": 1.48,
2040
+ "step": 282
2041
+ },
2042
+ {
2043
+ "epoch": 0.7894002789400278,
2044
+ "grad_norm": 1.4626745792933868,
2045
+ "learning_rate": 1.4314895854705641e-05,
2046
+ "loss": 1.4765,
2047
+ "step": 283
2048
+ },
2049
+ {
2050
+ "epoch": 0.7921896792189679,
2051
+ "grad_norm": 1.2910417535844496,
2052
+ "learning_rate": 1.4272925045661583e-05,
2053
+ "loss": 1.419,
2054
+ "step": 284
2055
+ },
2056
+ {
2057
+ "epoch": 0.7949790794979079,
2058
+ "grad_norm": 1.2373977812672998,
2059
+ "learning_rate": 1.4230861951674914e-05,
2060
+ "loss": 1.2251,
2061
+ "step": 285
2062
+ },
2063
+ {
2064
+ "epoch": 0.797768479776848,
2065
+ "grad_norm": 1.2785831952874724,
2066
+ "learning_rate": 1.4188707481207677e-05,
2067
+ "loss": 1.3471,
2068
+ "step": 286
2069
+ },
2070
+ {
2071
+ "epoch": 0.800557880055788,
2072
+ "grad_norm": 1.3683445048689633,
2073
+ "learning_rate": 1.4146462544695428e-05,
2074
+ "loss": 1.425,
2075
+ "step": 287
2076
+ },
2077
+ {
2078
+ "epoch": 0.803347280334728,
2079
+ "grad_norm": 1.1296732873251845,
2080
+ "learning_rate": 1.410412805452757e-05,
2081
+ "loss": 1.2578,
2082
+ "step": 288
2083
+ },
2084
+ {
2085
+ "epoch": 0.806136680613668,
2086
+ "grad_norm": 1.0829712122726374,
2087
+ "learning_rate": 1.4061704925027653e-05,
2088
+ "loss": 1.1799,
2089
+ "step": 289
2090
+ },
2091
+ {
2092
+ "epoch": 0.8089260808926081,
2093
+ "grad_norm": 1.3309074955401459,
2094
+ "learning_rate": 1.4019194072433626e-05,
2095
+ "loss": 1.1379,
2096
+ "step": 290
2097
+ },
2098
+ {
2099
+ "epoch": 0.8117154811715481,
2100
+ "grad_norm": 1.2343728023741232,
2101
+ "learning_rate": 1.3976596414878044e-05,
2102
+ "loss": 1.2411,
2103
+ "step": 291
2104
+ },
2105
+ {
2106
+ "epoch": 0.8145048814504882,
2107
+ "grad_norm": 1.214883458073917,
2108
+ "learning_rate": 1.393391287236824e-05,
2109
+ "loss": 1.3405,
2110
+ "step": 292
2111
+ },
2112
+ {
2113
+ "epoch": 0.8172942817294282,
2114
+ "grad_norm": 1.3091690676466778,
2115
+ "learning_rate": 1.3891144366766457e-05,
2116
+ "loss": 1.2606,
2117
+ "step": 293
2118
+ },
2119
+ {
2120
+ "epoch": 0.8200836820083682,
2121
+ "grad_norm": 13.631918339692971,
2122
+ "learning_rate": 1.3848291821769944e-05,
2123
+ "loss": 1.656,
2124
+ "step": 294
2125
+ },
2126
+ {
2127
+ "epoch": 0.8228730822873083,
2128
+ "grad_norm": 1.1657018881567924,
2129
+ "learning_rate": 1.380535616289099e-05,
2130
+ "loss": 1.2156,
2131
+ "step": 295
2132
+ },
2133
+ {
2134
+ "epoch": 0.8256624825662483,
2135
+ "grad_norm": 1.2347632552018992,
2136
+ "learning_rate": 1.3762338317436948e-05,
2137
+ "loss": 1.2669,
2138
+ "step": 296
2139
+ },
2140
+ {
2141
+ "epoch": 0.8284518828451883,
2142
+ "grad_norm": 1.4274262176368255,
2143
+ "learning_rate": 1.3719239214490203e-05,
2144
+ "loss": 1.3072,
2145
+ "step": 297
2146
+ },
2147
+ {
2148
+ "epoch": 0.8312412831241283,
2149
+ "grad_norm": 1.2894927915175798,
2150
+ "learning_rate": 1.3676059784888112e-05,
2151
+ "loss": 1.4573,
2152
+ "step": 298
2153
+ },
2154
+ {
2155
+ "epoch": 0.8340306834030683,
2156
+ "grad_norm": 1.4646947685259242,
2157
+ "learning_rate": 1.363280096120289e-05,
2158
+ "loss": 1.4956,
2159
+ "step": 299
2160
+ },
2161
+ {
2162
+ "epoch": 0.8368200836820083,
2163
+ "grad_norm": 1.2835331667441359,
2164
+ "learning_rate": 1.358946367772148e-05,
2165
+ "loss": 1.3713,
2166
+ "step": 300
2167
+ },
2168
+ {
2169
+ "epoch": 0.8396094839609484,
2170
+ "grad_norm": 1.3024578071081172,
2171
+ "learning_rate": 1.3546048870425356e-05,
2172
+ "loss": 1.345,
2173
+ "step": 301
2174
+ },
2175
+ {
2176
+ "epoch": 0.8423988842398884,
2177
+ "grad_norm": 1.144812031373234,
2178
+ "learning_rate": 1.3502557476970339e-05,
2179
+ "loss": 1.3447,
2180
+ "step": 302
2181
+ },
2182
+ {
2183
+ "epoch": 0.8451882845188284,
2184
+ "grad_norm": 1.255722753075089,
2185
+ "learning_rate": 1.3458990436666313e-05,
2186
+ "loss": 1.5169,
2187
+ "step": 303
2188
+ },
2189
+ {
2190
+ "epoch": 0.8479776847977685,
2191
+ "grad_norm": 1.1632268233137957,
2192
+ "learning_rate": 1.3415348690456955e-05,
2193
+ "loss": 1.289,
2194
+ "step": 304
2195
+ },
2196
+ {
2197
+ "epoch": 0.8507670850767085,
2198
+ "grad_norm": 1.2357758677374722,
2199
+ "learning_rate": 1.3371633180899417e-05,
2200
+ "loss": 1.4063,
2201
+ "step": 305
2202
+ },
2203
+ {
2204
+ "epoch": 0.8535564853556485,
2205
+ "grad_norm": 1.2854100609241097,
2206
+ "learning_rate": 1.3327844852143956e-05,
2207
+ "loss": 1.3677,
2208
+ "step": 306
2209
+ },
2210
+ {
2211
+ "epoch": 0.8563458856345886,
2212
+ "grad_norm": 1.2395591163738042,
2213
+ "learning_rate": 1.3283984649913552e-05,
2214
+ "loss": 1.3656,
2215
+ "step": 307
2216
+ },
2217
+ {
2218
+ "epoch": 0.8591352859135286,
2219
+ "grad_norm": 1.309692041297723,
2220
+ "learning_rate": 1.3240053521483483e-05,
2221
+ "loss": 1.3191,
2222
+ "step": 308
2223
+ },
2224
+ {
2225
+ "epoch": 0.8619246861924686,
2226
+ "grad_norm": 1.306594190283498,
2227
+ "learning_rate": 1.3196052415660856e-05,
2228
+ "loss": 1.2964,
2229
+ "step": 309
2230
+ },
2231
+ {
2232
+ "epoch": 0.8647140864714087,
2233
+ "grad_norm": 1.2332822767216993,
2234
+ "learning_rate": 1.3151982282764126e-05,
2235
+ "loss": 1.3172,
2236
+ "step": 310
2237
+ },
2238
+ {
2239
+ "epoch": 0.8675034867503487,
2240
+ "grad_norm": 1.2248947416069507,
2241
+ "learning_rate": 1.3107844074602566e-05,
2242
+ "loss": 1.2215,
2243
+ "step": 311
2244
+ },
2245
+ {
2246
+ "epoch": 0.8702928870292888,
2247
+ "grad_norm": 1.0883399467851869,
2248
+ "learning_rate": 1.3063638744455713e-05,
2249
+ "loss": 1.2487,
2250
+ "step": 312
2251
+ },
2252
+ {
2253
+ "epoch": 0.8730822873082287,
2254
+ "grad_norm": 1.3561694142459044,
2255
+ "learning_rate": 1.3019367247052781e-05,
2256
+ "loss": 1.5192,
2257
+ "step": 313
2258
+ },
2259
+ {
2260
+ "epoch": 0.8758716875871687,
2261
+ "grad_norm": 1.2195540202949509,
2262
+ "learning_rate": 1.297503053855203e-05,
2263
+ "loss": 1.315,
2264
+ "step": 314
2265
+ },
2266
+ {
2267
+ "epoch": 0.8786610878661087,
2268
+ "grad_norm": 1.2398989428685936,
2269
+ "learning_rate": 1.2930629576520133e-05,
2270
+ "loss": 1.2466,
2271
+ "step": 315
2272
+ },
2273
+ {
2274
+ "epoch": 0.8786610878661087,
2275
+ "eval_loss": 1.590497612953186,
2276
+ "eval_runtime": 158.7725,
2277
+ "eval_samples_per_second": 2.003,
2278
+ "eval_steps_per_second": 0.126,
2279
+ "step": 315
2280
+ },
2281
+ {
2282
+ "epoch": 0.8814504881450488,
2283
+ "grad_norm": 1.1820530696430398,
2284
+ "learning_rate": 1.2886165319911474e-05,
2285
+ "loss": 1.3187,
2286
+ "step": 316
2287
+ },
2288
+ {
2289
+ "epoch": 0.8842398884239888,
2290
+ "grad_norm": 1.2411796795161567,
2291
+ "learning_rate": 1.2841638729047463e-05,
2292
+ "loss": 1.3681,
2293
+ "step": 317
2294
+ },
2295
+ {
2296
+ "epoch": 0.8870292887029289,
2297
+ "grad_norm": 1.2561131847214293,
2298
+ "learning_rate": 1.2797050765595767e-05,
2299
+ "loss": 1.2601,
2300
+ "step": 318
2301
+ },
2302
+ {
2303
+ "epoch": 0.8898186889818689,
2304
+ "grad_norm": 1.415729283991149,
2305
+ "learning_rate": 1.2752402392549556e-05,
2306
+ "loss": 1.4896,
2307
+ "step": 319
2308
+ },
2309
+ {
2310
+ "epoch": 0.8926080892608089,
2311
+ "grad_norm": 1.3261631641702796,
2312
+ "learning_rate": 1.2707694574206712e-05,
2313
+ "loss": 1.5607,
2314
+ "step": 320
2315
+ },
2316
+ {
2317
+ "epoch": 0.895397489539749,
2318
+ "grad_norm": 1.240314237616232,
2319
+ "learning_rate": 1.2662928276148985e-05,
2320
+ "loss": 1.3429,
2321
+ "step": 321
2322
+ },
2323
+ {
2324
+ "epoch": 0.898186889818689,
2325
+ "grad_norm": 1.3000617526029077,
2326
+ "learning_rate": 1.2618104465221148e-05,
2327
+ "loss": 1.4272,
2328
+ "step": 322
2329
+ },
2330
+ {
2331
+ "epoch": 0.900976290097629,
2332
+ "grad_norm": 1.16464364915601,
2333
+ "learning_rate": 1.2573224109510112e-05,
2334
+ "loss": 1.2306,
2335
+ "step": 323
2336
+ },
2337
+ {
2338
+ "epoch": 0.9037656903765691,
2339
+ "grad_norm": 1.2109886144147917,
2340
+ "learning_rate": 1.2528288178324029e-05,
2341
+ "loss": 1.353,
2342
+ "step": 324
2343
+ },
2344
+ {
2345
+ "epoch": 0.9065550906555091,
2346
+ "grad_norm": 1.1714274928342496,
2347
+ "learning_rate": 1.2483297642171332e-05,
2348
+ "loss": 1.2997,
2349
+ "step": 325
2350
+ },
2351
+ {
2352
+ "epoch": 0.9093444909344491,
2353
+ "grad_norm": 1.1814072235776383,
2354
+ "learning_rate": 1.2438253472739805e-05,
2355
+ "loss": 1.3144,
2356
+ "step": 326
2357
+ },
2358
+ {
2359
+ "epoch": 0.9121338912133892,
2360
+ "grad_norm": 1.3461681399899779,
2361
+ "learning_rate": 1.2393156642875579e-05,
2362
+ "loss": 1.3618,
2363
+ "step": 327
2364
+ },
2365
+ {
2366
+ "epoch": 0.9149232914923291,
2367
+ "grad_norm": 1.456924757382429,
2368
+ "learning_rate": 1.234800812656212e-05,
2369
+ "loss": 1.3796,
2370
+ "step": 328
2371
+ },
2372
+ {
2373
+ "epoch": 0.9177126917712691,
2374
+ "grad_norm": 1.2321016000046479,
2375
+ "learning_rate": 1.23028088988992e-05,
2376
+ "loss": 1.3771,
2377
+ "step": 329
2378
+ },
2379
+ {
2380
+ "epoch": 0.9205020920502092,
2381
+ "grad_norm": 1.2405013760966828,
2382
+ "learning_rate": 1.2257559936081833e-05,
2383
+ "loss": 1.2744,
2384
+ "step": 330
2385
+ },
2386
+ {
2387
+ "epoch": 0.9232914923291492,
2388
+ "grad_norm": 1.2677524234506234,
2389
+ "learning_rate": 1.2212262215379199e-05,
2390
+ "loss": 1.3578,
2391
+ "step": 331
2392
+ },
2393
+ {
2394
+ "epoch": 0.9260808926080892,
2395
+ "grad_norm": 1.2230537842102542,
2396
+ "learning_rate": 1.2166916715113521e-05,
2397
+ "loss": 1.4323,
2398
+ "step": 332
2399
+ },
2400
+ {
2401
+ "epoch": 0.9288702928870293,
2402
+ "grad_norm": 1.2690522272930789,
2403
+ "learning_rate": 1.2121524414638958e-05,
2404
+ "loss": 1.2845,
2405
+ "step": 333
2406
+ },
2407
+ {
2408
+ "epoch": 0.9316596931659693,
2409
+ "grad_norm": 1.2822697397851108,
2410
+ "learning_rate": 1.207608629432043e-05,
2411
+ "loss": 1.6128,
2412
+ "step": 334
2413
+ },
2414
+ {
2415
+ "epoch": 0.9344490934449093,
2416
+ "grad_norm": 1.2997575387211082,
2417
+ "learning_rate": 1.2030603335512467e-05,
2418
+ "loss": 1.3401,
2419
+ "step": 335
2420
+ },
2421
+ {
2422
+ "epoch": 0.9372384937238494,
2423
+ "grad_norm": 1.5059463142396166,
2424
+ "learning_rate": 1.1985076520537995e-05,
2425
+ "loss": 1.2793,
2426
+ "step": 336
2427
+ },
2428
+ {
2429
+ "epoch": 0.9400278940027894,
2430
+ "grad_norm": 1.5118948746263785,
2431
+ "learning_rate": 1.1939506832667129e-05,
2432
+ "loss": 1.3152,
2433
+ "step": 337
2434
+ },
2435
+ {
2436
+ "epoch": 0.9428172942817294,
2437
+ "grad_norm": 1.2816385360112317,
2438
+ "learning_rate": 1.1893895256095937e-05,
2439
+ "loss": 1.4573,
2440
+ "step": 338
2441
+ },
2442
+ {
2443
+ "epoch": 0.9456066945606695,
2444
+ "grad_norm": 1.1973723679649781,
2445
+ "learning_rate": 1.1848242775925188e-05,
2446
+ "loss": 1.1473,
2447
+ "step": 339
2448
+ },
2449
+ {
2450
+ "epoch": 0.9483960948396095,
2451
+ "grad_norm": 1.157556800970344,
2452
+ "learning_rate": 1.180255037813906e-05,
2453
+ "loss": 1.362,
2454
+ "step": 340
2455
+ },
2456
+ {
2457
+ "epoch": 0.9511854951185496,
2458
+ "grad_norm": 1.3366940604017197,
2459
+ "learning_rate": 1.1756819049583861e-05,
2460
+ "loss": 1.5201,
2461
+ "step": 341
2462
+ },
2463
+ {
2464
+ "epoch": 0.9539748953974896,
2465
+ "grad_norm": 1.396951726353029,
2466
+ "learning_rate": 1.1711049777946717e-05,
2467
+ "loss": 1.4882,
2468
+ "step": 342
2469
+ },
2470
+ {
2471
+ "epoch": 0.9567642956764296,
2472
+ "grad_norm": 1.1789267346275334,
2473
+ "learning_rate": 1.166524355173422e-05,
2474
+ "loss": 1.3806,
2475
+ "step": 343
2476
+ },
2477
+ {
2478
+ "epoch": 0.9595536959553695,
2479
+ "grad_norm": 1.2075413726006268,
2480
+ "learning_rate": 1.1619401360251104e-05,
2481
+ "loss": 1.2451,
2482
+ "step": 344
2483
+ },
2484
+ {
2485
+ "epoch": 0.9623430962343096,
2486
+ "grad_norm": 1.2379647427390967,
2487
+ "learning_rate": 1.1573524193578863e-05,
2488
+ "loss": 1.1581,
2489
+ "step": 345
2490
+ },
2491
+ {
2492
+ "epoch": 0.9651324965132496,
2493
+ "grad_norm": 1.1953475180567759,
2494
+ "learning_rate": 1.1527613042554368e-05,
2495
+ "loss": 1.3765,
2496
+ "step": 346
2497
+ },
2498
+ {
2499
+ "epoch": 0.9679218967921897,
2500
+ "grad_norm": 1.2657967489739972,
2501
+ "learning_rate": 1.1481668898748474e-05,
2502
+ "loss": 1.335,
2503
+ "step": 347
2504
+ },
2505
+ {
2506
+ "epoch": 0.9707112970711297,
2507
+ "grad_norm": 1.2590876717921935,
2508
+ "learning_rate": 1.1435692754444604e-05,
2509
+ "loss": 1.339,
2510
+ "step": 348
2511
+ },
2512
+ {
2513
+ "epoch": 0.9735006973500697,
2514
+ "grad_norm": 1.3430351369524687,
2515
+ "learning_rate": 1.1389685602617302e-05,
2516
+ "loss": 1.3618,
2517
+ "step": 349
2518
+ },
2519
+ {
2520
+ "epoch": 0.9762900976290098,
2521
+ "grad_norm": 1.216928490077423,
2522
+ "learning_rate": 1.134364843691082e-05,
2523
+ "loss": 1.4504,
2524
+ "step": 350
2525
+ },
2526
+ {
2527
+ "epoch": 0.9790794979079498,
2528
+ "grad_norm": 1.2961172668976244,
2529
+ "learning_rate": 1.1297582251617618e-05,
2530
+ "loss": 1.3752,
2531
+ "step": 351
2532
+ },
2533
+ {
2534
+ "epoch": 0.9818688981868898,
2535
+ "grad_norm": 1.3163958626398011,
2536
+ "learning_rate": 1.1251488041656929e-05,
2537
+ "loss": 1.2906,
2538
+ "step": 352
2539
+ },
2540
+ {
2541
+ "epoch": 0.9846582984658299,
2542
+ "grad_norm": 1.2499583055181795,
2543
+ "learning_rate": 1.1205366802553231e-05,
2544
+ "loss": 1.3641,
2545
+ "step": 353
2546
+ },
2547
+ {
2548
+ "epoch": 0.9874476987447699,
2549
+ "grad_norm": 1.5088593864499633,
2550
+ "learning_rate": 1.1159219530414786e-05,
2551
+ "loss": 1.4534,
2552
+ "step": 354
2553
+ },
2554
+ {
2555
+ "epoch": 0.9902370990237099,
2556
+ "grad_norm": 1.1931538459250062,
2557
+ "learning_rate": 1.1113047221912097e-05,
2558
+ "loss": 1.3285,
2559
+ "step": 355
2560
+ },
2561
+ {
2562
+ "epoch": 0.99302649930265,
2563
+ "grad_norm": 1.1763206233025147,
2564
+ "learning_rate": 1.1066850874256387e-05,
2565
+ "loss": 1.3928,
2566
+ "step": 356
2567
+ },
2568
+ {
2569
+ "epoch": 0.99581589958159,
2570
+ "grad_norm": 1.2398324928784012,
2571
+ "learning_rate": 1.1020631485178084e-05,
2572
+ "loss": 1.4118,
2573
+ "step": 357
2574
+ },
2575
+ {
2576
+ "epoch": 0.99860529986053,
2577
+ "grad_norm": 1.1658511676647905,
2578
+ "learning_rate": 1.0974390052905238e-05,
2579
+ "loss": 1.2735,
2580
+ "step": 358
2581
+ },
2582
+ {
2583
+ "epoch": 1.0,
2584
+ "grad_norm": 1.1658511676647905,
2585
+ "learning_rate": 1.0928127576141992e-05,
2586
+ "loss": 0.631,
2587
+ "step": 359
2588
+ },
2589
+ {
2590
+ "epoch": 1.00278940027894,
2591
+ "grad_norm": 1.2279956840576842,
2592
+ "learning_rate": 1.0881845054046988e-05,
2593
+ "loss": 1.3144,
2594
+ "step": 360
2595
+ },
2596
+ {
2597
+ "epoch": 1.00278940027894,
2598
+ "eval_loss": 1.5843528509140015,
2599
+ "eval_runtime": 158.9441,
2600
+ "eval_samples_per_second": 2.001,
2601
+ "eval_steps_per_second": 0.126,
2602
+ "step": 360
2603
+ },
2604
+ {
2605
+ "epoch": 1.00557880055788,
2606
+ "grad_norm": 1.122956387667915,
2607
+ "learning_rate": 1.0835543486211815e-05,
2608
+ "loss": 1.1875,
2609
+ "step": 361
2610
+ },
2611
+ {
2612
+ "epoch": 1.00836820083682,
2613
+ "grad_norm": 1.1487145048815297,
2614
+ "learning_rate": 1.0789223872639385e-05,
2615
+ "loss": 1.3283,
2616
+ "step": 362
2617
+ },
2618
+ {
2619
+ "epoch": 1.0111576011157601,
2620
+ "grad_norm": 1.210362167885254,
2621
+ "learning_rate": 1.0742887213722372e-05,
2622
+ "loss": 1.1607,
2623
+ "step": 363
2624
+ },
2625
+ {
2626
+ "epoch": 1.0139470013947,
2627
+ "grad_norm": 1.167683710016959,
2628
+ "learning_rate": 1.0696534510221575e-05,
2629
+ "loss": 1.3125,
2630
+ "step": 364
2631
+ },
2632
+ {
2633
+ "epoch": 1.0167364016736402,
2634
+ "grad_norm": 1.224749951044779,
2635
+ "learning_rate": 1.065016676324433e-05,
2636
+ "loss": 1.2384,
2637
+ "step": 365
2638
+ },
2639
+ {
2640
+ "epoch": 1.0195258019525801,
2641
+ "grad_norm": 1.1566624177175886,
2642
+ "learning_rate": 1.0603784974222862e-05,
2643
+ "loss": 1.3956,
2644
+ "step": 366
2645
+ },
2646
+ {
2647
+ "epoch": 1.0223152022315203,
2648
+ "grad_norm": 1.2969848771007706,
2649
+ "learning_rate": 1.0557390144892684e-05,
2650
+ "loss": 1.3216,
2651
+ "step": 367
2652
+ },
2653
+ {
2654
+ "epoch": 1.0251046025104602,
2655
+ "grad_norm": 1.1557275837790277,
2656
+ "learning_rate": 1.0510983277270945e-05,
2657
+ "loss": 1.1656,
2658
+ "step": 368
2659
+ },
2660
+ {
2661
+ "epoch": 1.0278940027894004,
2662
+ "grad_norm": 1.1274495156209017,
2663
+ "learning_rate": 1.0464565373634784e-05,
2664
+ "loss": 1.234,
2665
+ "step": 369
2666
+ },
2667
+ {
2668
+ "epoch": 1.0306834030683403,
2669
+ "grad_norm": 1.2136920982158583,
2670
+ "learning_rate": 1.0418137436499699e-05,
2671
+ "loss": 1.1579,
2672
+ "step": 370
2673
+ },
2674
+ {
2675
+ "epoch": 1.0334728033472804,
2676
+ "grad_norm": 1.2136361742646686,
2677
+ "learning_rate": 1.0371700468597886e-05,
2678
+ "loss": 1.3869,
2679
+ "step": 371
2680
+ },
2681
+ {
2682
+ "epoch": 1.0362622036262203,
2683
+ "grad_norm": 1.1175097910575362,
2684
+ "learning_rate": 1.0325255472856586e-05,
2685
+ "loss": 1.214,
2686
+ "step": 372
2687
+ },
2688
+ {
2689
+ "epoch": 1.0390516039051605,
2690
+ "grad_norm": 1.1395096520260177,
2691
+ "learning_rate": 1.0278803452376416e-05,
2692
+ "loss": 1.436,
2693
+ "step": 373
2694
+ },
2695
+ {
2696
+ "epoch": 1.0418410041841004,
2697
+ "grad_norm": 1.1082860671382055,
2698
+ "learning_rate": 1.0232345410409718e-05,
2699
+ "loss": 1.2397,
2700
+ "step": 374
2701
+ },
2702
+ {
2703
+ "epoch": 1.0446304044630403,
2704
+ "grad_norm": 1.1045608548981312,
2705
+ "learning_rate": 1.018588235033888e-05,
2706
+ "loss": 1.2764,
2707
+ "step": 375
2708
+ },
2709
+ {
2710
+ "epoch": 1.0474198047419805,
2711
+ "grad_norm": 1.1792908172632213,
2712
+ "learning_rate": 1.0139415275654671e-05,
2713
+ "loss": 1.3103,
2714
+ "step": 376
2715
+ },
2716
+ {
2717
+ "epoch": 1.0502092050209204,
2718
+ "grad_norm": 1.1928217784395125,
2719
+ "learning_rate": 1.0092945189934558e-05,
2720
+ "loss": 1.2358,
2721
+ "step": 377
2722
+ },
2723
+ {
2724
+ "epoch": 1.0529986052998606,
2725
+ "grad_norm": 1.2145452681867133,
2726
+ "learning_rate": 1.0046473096821047e-05,
2727
+ "loss": 1.196,
2728
+ "step": 378
2729
+ },
2730
+ {
2731
+ "epoch": 1.0557880055788005,
2732
+ "grad_norm": 1.2041436209676546,
2733
+ "learning_rate": 1e-05,
2734
+ "loss": 1.3178,
2735
+ "step": 379
2736
+ },
2737
+ {
2738
+ "epoch": 1.0585774058577406,
2739
+ "grad_norm": 1.148399751524864,
2740
+ "learning_rate": 9.953526903178956e-06,
2741
+ "loss": 1.4499,
2742
+ "step": 380
2743
+ },
2744
+ {
2745
+ "epoch": 1.0613668061366806,
2746
+ "grad_norm": 1.1974026865483098,
2747
+ "learning_rate": 9.907054810065446e-06,
2748
+ "loss": 1.2057,
2749
+ "step": 381
2750
+ },
2751
+ {
2752
+ "epoch": 1.0641562064156207,
2753
+ "grad_norm": 1.5812229698878382,
2754
+ "learning_rate": 9.860584724345334e-06,
2755
+ "loss": 1.482,
2756
+ "step": 382
2757
+ },
2758
+ {
2759
+ "epoch": 1.0669456066945606,
2760
+ "grad_norm": 1.1834008069729351,
2761
+ "learning_rate": 9.81411764966112e-06,
2762
+ "loss": 1.3193,
2763
+ "step": 383
2764
+ },
2765
+ {
2766
+ "epoch": 1.0697350069735008,
2767
+ "grad_norm": 1.0680962669389744,
2768
+ "learning_rate": 9.767654589590284e-06,
2769
+ "loss": 1.1936,
2770
+ "step": 384
2771
+ },
2772
+ {
2773
+ "epoch": 1.0725244072524407,
2774
+ "grad_norm": 1.2433855122802504,
2775
+ "learning_rate": 9.721196547623585e-06,
2776
+ "loss": 1.3485,
2777
+ "step": 385
2778
+ },
2779
+ {
2780
+ "epoch": 1.0753138075313808,
2781
+ "grad_norm": 1.1818473383563528,
2782
+ "learning_rate": 9.674744527143419e-06,
2783
+ "loss": 1.2435,
2784
+ "step": 386
2785
+ },
2786
+ {
2787
+ "epoch": 1.0781032078103208,
2788
+ "grad_norm": 1.1606102472129574,
2789
+ "learning_rate": 9.628299531402118e-06,
2790
+ "loss": 1.352,
2791
+ "step": 387
2792
+ },
2793
+ {
2794
+ "epoch": 1.080892608089261,
2795
+ "grad_norm": 1.3474388049187347,
2796
+ "learning_rate": 9.581862563500305e-06,
2797
+ "loss": 1.3987,
2798
+ "step": 388
2799
+ },
2800
+ {
2801
+ "epoch": 1.0836820083682008,
2802
+ "grad_norm": 1.2419068969830154,
2803
+ "learning_rate": 9.535434626365221e-06,
2804
+ "loss": 1.3226,
2805
+ "step": 389
2806
+ },
2807
+ {
2808
+ "epoch": 1.086471408647141,
2809
+ "grad_norm": 1.1382928547735685,
2810
+ "learning_rate": 9.489016722729059e-06,
2811
+ "loss": 1.2358,
2812
+ "step": 390
2813
+ },
2814
+ {
2815
+ "epoch": 1.089260808926081,
2816
+ "grad_norm": 1.062780531568017,
2817
+ "learning_rate": 9.442609855107317e-06,
2818
+ "loss": 1.1819,
2819
+ "step": 391
2820
+ },
2821
+ {
2822
+ "epoch": 1.0920502092050208,
2823
+ "grad_norm": 1.10859906931155,
2824
+ "learning_rate": 9.39621502577714e-06,
2825
+ "loss": 1.2527,
2826
+ "step": 392
2827
+ },
2828
+ {
2829
+ "epoch": 1.094839609483961,
2830
+ "grad_norm": 1.2185757174534009,
2831
+ "learning_rate": 9.349833236755675e-06,
2832
+ "loss": 1.3156,
2833
+ "step": 393
2834
+ },
2835
+ {
2836
+ "epoch": 1.097629009762901,
2837
+ "grad_norm": 1.225759251550299,
2838
+ "learning_rate": 9.30346548977843e-06,
2839
+ "loss": 1.1902,
2840
+ "step": 394
2841
+ },
2842
+ {
2843
+ "epoch": 1.100418410041841,
2844
+ "grad_norm": 1.143304170948198,
2845
+ "learning_rate": 9.257112786277631e-06,
2846
+ "loss": 1.6164,
2847
+ "step": 395
2848
+ },
2849
+ {
2850
+ "epoch": 1.103207810320781,
2851
+ "grad_norm": 1.1929513674684669,
2852
+ "learning_rate": 9.21077612736062e-06,
2853
+ "loss": 1.3059,
2854
+ "step": 396
2855
+ },
2856
+ {
2857
+ "epoch": 1.1059972105997211,
2858
+ "grad_norm": 1.2652518498481602,
2859
+ "learning_rate": 9.164456513788186e-06,
2860
+ "loss": 1.3585,
2861
+ "step": 397
2862
+ },
2863
+ {
2864
+ "epoch": 1.108786610878661,
2865
+ "grad_norm": 1.181284519624733,
2866
+ "learning_rate": 9.118154945953015e-06,
2867
+ "loss": 1.3096,
2868
+ "step": 398
2869
+ },
2870
+ {
2871
+ "epoch": 1.1115760111576012,
2872
+ "grad_norm": 1.1391566905227906,
2873
+ "learning_rate": 9.07187242385801e-06,
2874
+ "loss": 1.2707,
2875
+ "step": 399
2876
+ },
2877
+ {
2878
+ "epoch": 1.114365411436541,
2879
+ "grad_norm": 1.0825559641774511,
2880
+ "learning_rate": 9.025609947094765e-06,
2881
+ "loss": 1.2668,
2882
+ "step": 400
2883
+ },
2884
+ {
2885
+ "epoch": 1.1171548117154813,
2886
+ "grad_norm": 1.1830262642050864,
2887
+ "learning_rate": 8.979368514821917e-06,
2888
+ "loss": 1.5181,
2889
+ "step": 401
2890
+ },
2891
+ {
2892
+ "epoch": 1.1199442119944212,
2893
+ "grad_norm": 1.3026726939018842,
2894
+ "learning_rate": 8.933149125743615e-06,
2895
+ "loss": 1.219,
2896
+ "step": 402
2897
+ },
2898
+ {
2899
+ "epoch": 1.1227336122733613,
2900
+ "grad_norm": 1.144101939773731,
2901
+ "learning_rate": 8.88695277808791e-06,
2902
+ "loss": 1.3683,
2903
+ "step": 403
2904
+ },
2905
+ {
2906
+ "epoch": 1.1255230125523012,
2907
+ "grad_norm": 1.2305348777008256,
2908
+ "learning_rate": 8.840780469585215e-06,
2909
+ "loss": 1.3366,
2910
+ "step": 404
2911
+ },
2912
+ {
2913
+ "epoch": 1.1283124128312414,
2914
+ "grad_norm": 1.209754151387273,
2915
+ "learning_rate": 8.79463319744677e-06,
2916
+ "loss": 1.1868,
2917
+ "step": 405
2918
+ },
2919
+ {
2920
+ "epoch": 1.1283124128312414,
2921
+ "eval_loss": 1.5783870220184326,
2922
+ "eval_runtime": 158.6338,
2923
+ "eval_samples_per_second": 2.005,
2924
+ "eval_steps_per_second": 0.126,
2925
+ "step": 405
2926
+ },
2927
+ {
2928
+ "epoch": 1.1311018131101813,
2929
+ "grad_norm": 1.1895020228803659,
2930
+ "learning_rate": 8.748511958343076e-06,
2931
+ "loss": 1.2254,
2932
+ "step": 406
2933
+ },
2934
+ {
2935
+ "epoch": 1.1338912133891212,
2936
+ "grad_norm": 1.0698636977312026,
2937
+ "learning_rate": 8.702417748382384e-06,
2938
+ "loss": 1.2746,
2939
+ "step": 407
2940
+ },
2941
+ {
2942
+ "epoch": 1.1366806136680614,
2943
+ "grad_norm": 1.1610395502416724,
2944
+ "learning_rate": 8.656351563089185e-06,
2945
+ "loss": 1.1515,
2946
+ "step": 408
2947
+ },
2948
+ {
2949
+ "epoch": 1.1394700139470013,
2950
+ "grad_norm": 1.0723262933707627,
2951
+ "learning_rate": 8.610314397382701e-06,
2952
+ "loss": 1.2238,
2953
+ "step": 409
2954
+ },
2955
+ {
2956
+ "epoch": 1.1422594142259415,
2957
+ "grad_norm": 1.1092261729942694,
2958
+ "learning_rate": 8.564307245555403e-06,
2959
+ "loss": 1.2052,
2960
+ "step": 410
2961
+ },
2962
+ {
2963
+ "epoch": 1.1450488145048814,
2964
+ "grad_norm": 1.0779129863872163,
2965
+ "learning_rate": 8.51833110125153e-06,
2966
+ "loss": 1.3471,
2967
+ "step": 411
2968
+ },
2969
+ {
2970
+ "epoch": 1.1478382147838215,
2971
+ "grad_norm": 1.2861746029314869,
2972
+ "learning_rate": 8.472386957445635e-06,
2973
+ "loss": 1.2837,
2974
+ "step": 412
2975
+ },
2976
+ {
2977
+ "epoch": 1.1506276150627615,
2978
+ "grad_norm": 1.0742534390601233,
2979
+ "learning_rate": 8.426475806421139e-06,
2980
+ "loss": 1.3086,
2981
+ "step": 413
2982
+ },
2983
+ {
2984
+ "epoch": 1.1534170153417016,
2985
+ "grad_norm": 1.1872147964682518,
2986
+ "learning_rate": 8.380598639748898e-06,
2987
+ "loss": 1.2254,
2988
+ "step": 414
2989
+ },
2990
+ {
2991
+ "epoch": 1.1562064156206415,
2992
+ "grad_norm": 1.07165806884352,
2993
+ "learning_rate": 8.334756448265782e-06,
2994
+ "loss": 1.275,
2995
+ "step": 415
2996
+ },
2997
+ {
2998
+ "epoch": 1.1589958158995817,
2999
+ "grad_norm": 1.2751759485563616,
3000
+ "learning_rate": 8.288950222053287e-06,
3001
+ "loss": 1.1922,
3002
+ "step": 416
3003
+ },
3004
+ {
3005
+ "epoch": 1.1617852161785216,
3006
+ "grad_norm": 1.1196667842580768,
3007
+ "learning_rate": 8.243180950416142e-06,
3008
+ "loss": 1.2673,
3009
+ "step": 417
3010
+ },
3011
+ {
3012
+ "epoch": 1.1645746164574617,
3013
+ "grad_norm": 1.1440592720887446,
3014
+ "learning_rate": 8.197449621860944e-06,
3015
+ "loss": 1.3993,
3016
+ "step": 418
3017
+ },
3018
+ {
3019
+ "epoch": 1.1673640167364017,
3020
+ "grad_norm": 1.176436140811514,
3021
+ "learning_rate": 8.151757224074815e-06,
3022
+ "loss": 1.3844,
3023
+ "step": 419
3024
+ },
3025
+ {
3026
+ "epoch": 1.1701534170153418,
3027
+ "grad_norm": 1.2831755398630933,
3028
+ "learning_rate": 8.106104743904065e-06,
3029
+ "loss": 1.4343,
3030
+ "step": 420
3031
+ },
3032
+ {
3033
+ "epoch": 1.1729428172942817,
3034
+ "grad_norm": 1.181724274442928,
3035
+ "learning_rate": 8.060493167332874e-06,
3036
+ "loss": 1.3049,
3037
+ "step": 421
3038
+ },
3039
+ {
3040
+ "epoch": 1.1757322175732217,
3041
+ "grad_norm": 1.1464276027570777,
3042
+ "learning_rate": 8.014923479462007e-06,
3043
+ "loss": 1.234,
3044
+ "step": 422
3045
+ },
3046
+ {
3047
+ "epoch": 1.1785216178521618,
3048
+ "grad_norm": 1.1377604892058606,
3049
+ "learning_rate": 7.969396664487534e-06,
3050
+ "loss": 1.2889,
3051
+ "step": 423
3052
+ },
3053
+ {
3054
+ "epoch": 1.1813110181311017,
3055
+ "grad_norm": 1.1339458036648977,
3056
+ "learning_rate": 7.923913705679573e-06,
3057
+ "loss": 1.2602,
3058
+ "step": 424
3059
+ },
3060
+ {
3061
+ "epoch": 1.1841004184100419,
3062
+ "grad_norm": 1.0197395887525806,
3063
+ "learning_rate": 7.878475585361045e-06,
3064
+ "loss": 1.126,
3065
+ "step": 425
3066
+ },
3067
+ {
3068
+ "epoch": 1.1868898186889818,
3069
+ "grad_norm": 1.080702461827881,
3070
+ "learning_rate": 7.833083284886484e-06,
3071
+ "loss": 1.1639,
3072
+ "step": 426
3073
+ },
3074
+ {
3075
+ "epoch": 1.189679218967922,
3076
+ "grad_norm": 1.2607987099270583,
3077
+ "learning_rate": 7.787737784620803e-06,
3078
+ "loss": 1.4396,
3079
+ "step": 427
3080
+ },
3081
+ {
3082
+ "epoch": 1.1924686192468619,
3083
+ "grad_norm": 1.1254779807057225,
3084
+ "learning_rate": 7.74244006391817e-06,
3085
+ "loss": 1.3822,
3086
+ "step": 428
3087
+ },
3088
+ {
3089
+ "epoch": 1.195258019525802,
3090
+ "grad_norm": 1.1774004879920348,
3091
+ "learning_rate": 7.697191101100802e-06,
3092
+ "loss": 1.3326,
3093
+ "step": 429
3094
+ },
3095
+ {
3096
+ "epoch": 1.198047419804742,
3097
+ "grad_norm": 1.0896896006152201,
3098
+ "learning_rate": 7.651991873437884e-06,
3099
+ "loss": 1.2928,
3100
+ "step": 430
3101
+ },
3102
+ {
3103
+ "epoch": 1.200836820083682,
3104
+ "grad_norm": 1.1261304598713149,
3105
+ "learning_rate": 7.606843357124426e-06,
3106
+ "loss": 1.2123,
3107
+ "step": 431
3108
+ },
3109
+ {
3110
+ "epoch": 1.203626220362622,
3111
+ "grad_norm": 1.1658455006742379,
3112
+ "learning_rate": 7.561746527260197e-06,
3113
+ "loss": 1.3404,
3114
+ "step": 432
3115
+ },
3116
+ {
3117
+ "epoch": 1.2064156206415622,
3118
+ "grad_norm": 1.233442454888862,
3119
+ "learning_rate": 7.516702357828672e-06,
3120
+ "loss": 1.4484,
3121
+ "step": 433
3122
+ },
3123
+ {
3124
+ "epoch": 1.209205020920502,
3125
+ "grad_norm": 1.20274243777343,
3126
+ "learning_rate": 7.471711821675973e-06,
3127
+ "loss": 1.1722,
3128
+ "step": 434
3129
+ },
3130
+ {
3131
+ "epoch": 1.2119944211994422,
3132
+ "grad_norm": 1.0643983498668543,
3133
+ "learning_rate": 7.42677589048989e-06,
3134
+ "loss": 1.1699,
3135
+ "step": 435
3136
+ },
3137
+ {
3138
+ "epoch": 1.2147838214783822,
3139
+ "grad_norm": 1.0591582855658968,
3140
+ "learning_rate": 7.381895534778852e-06,
3141
+ "loss": 1.3417,
3142
+ "step": 436
3143
+ },
3144
+ {
3145
+ "epoch": 1.217573221757322,
3146
+ "grad_norm": 1.1587399832790362,
3147
+ "learning_rate": 7.337071723851018e-06,
3148
+ "loss": 1.2767,
3149
+ "step": 437
3150
+ },
3151
+ {
3152
+ "epoch": 1.2203626220362622,
3153
+ "grad_norm": 1.1559690690845343,
3154
+ "learning_rate": 7.292305425793291e-06,
3155
+ "loss": 1.3692,
3156
+ "step": 438
3157
+ },
3158
+ {
3159
+ "epoch": 1.2231520223152021,
3160
+ "grad_norm": 1.1477911804467655,
3161
+ "learning_rate": 7.247597607450446e-06,
3162
+ "loss": 1.4562,
3163
+ "step": 439
3164
+ },
3165
+ {
3166
+ "epoch": 1.2259414225941423,
3167
+ "grad_norm": 1.244415118958711,
3168
+ "learning_rate": 7.2029492344042395e-06,
3169
+ "loss": 1.3252,
3170
+ "step": 440
3171
+ },
3172
+ {
3173
+ "epoch": 1.2287308228730822,
3174
+ "grad_norm": 1.1256137359133935,
3175
+ "learning_rate": 7.1583612709525405e-06,
3176
+ "loss": 1.1368,
3177
+ "step": 441
3178
+ },
3179
+ {
3180
+ "epoch": 1.2315202231520224,
3181
+ "grad_norm": 1.1793196712059852,
3182
+ "learning_rate": 7.113834680088527e-06,
3183
+ "loss": 1.3781,
3184
+ "step": 442
3185
+ },
3186
+ {
3187
+ "epoch": 1.2343096234309623,
3188
+ "grad_norm": 1.2150025131043334,
3189
+ "learning_rate": 7.06937042347987e-06,
3190
+ "loss": 1.2111,
3191
+ "step": 443
3192
+ },
3193
+ {
3194
+ "epoch": 1.2370990237099024,
3195
+ "grad_norm": 1.1023846781294708,
3196
+ "learning_rate": 7.024969461447973e-06,
3197
+ "loss": 1.2601,
3198
+ "step": 444
3199
+ },
3200
+ {
3201
+ "epoch": 1.2398884239888424,
3202
+ "grad_norm": 1.0716238418458055,
3203
+ "learning_rate": 6.980632752947221e-06,
3204
+ "loss": 1.153,
3205
+ "step": 445
3206
+ },
3207
+ {
3208
+ "epoch": 1.2426778242677825,
3209
+ "grad_norm": 1.2507786838071617,
3210
+ "learning_rate": 6.936361255544288e-06,
3211
+ "loss": 1.2586,
3212
+ "step": 446
3213
+ },
3214
+ {
3215
+ "epoch": 1.2454672245467224,
3216
+ "grad_norm": 1.214628056363252,
3217
+ "learning_rate": 6.892155925397437e-06,
3218
+ "loss": 1.2378,
3219
+ "step": 447
3220
+ },
3221
+ {
3222
+ "epoch": 1.2482566248256626,
3223
+ "grad_norm": 1.1603500824157453,
3224
+ "learning_rate": 6.848017717235878e-06,
3225
+ "loss": 1.3192,
3226
+ "step": 448
3227
+ },
3228
+ {
3229
+ "epoch": 1.2510460251046025,
3230
+ "grad_norm": 1.1687329408420344,
3231
+ "learning_rate": 6.803947584339148e-06,
3232
+ "loss": 1.2094,
3233
+ "step": 449
3234
+ },
3235
+ {
3236
+ "epoch": 1.2538354253835426,
3237
+ "grad_norm": 1.2918425935100006,
3238
+ "learning_rate": 6.7599464785165195e-06,
3239
+ "loss": 1.3102,
3240
+ "step": 450
3241
+ },
3242
+ {
3243
+ "epoch": 1.2538354253835426,
3244
+ "eval_loss": 1.5750354528427124,
3245
+ "eval_runtime": 158.0557,
3246
+ "eval_samples_per_second": 2.012,
3247
+ "eval_steps_per_second": 0.127,
3248
+ "step": 450
3249
+ },
3250
+ {
3251
+ "epoch": 1.2566248256624826,
3252
+ "grad_norm": 1.2510681039370228,
3253
+ "learning_rate": 6.716015350086449e-06,
3254
+ "loss": 1.2365,
3255
+ "step": 451
3256
+ },
3257
+ {
3258
+ "epoch": 1.2594142259414225,
3259
+ "grad_norm": 1.1439803976132223,
3260
+ "learning_rate": 6.672155147856046e-06,
3261
+ "loss": 1.2992,
3262
+ "step": 452
3263
+ },
3264
+ {
3265
+ "epoch": 1.2622036262203626,
3266
+ "grad_norm": 1.2149245156903248,
3267
+ "learning_rate": 6.628366819100586e-06,
3268
+ "loss": 1.4374,
3269
+ "step": 453
3270
+ },
3271
+ {
3272
+ "epoch": 1.2649930264993028,
3273
+ "grad_norm": 1.1763769310735905,
3274
+ "learning_rate": 6.584651309543049e-06,
3275
+ "loss": 1.2003,
3276
+ "step": 454
3277
+ },
3278
+ {
3279
+ "epoch": 1.2677824267782427,
3280
+ "grad_norm": 1.127070003440945,
3281
+ "learning_rate": 6.54100956333369e-06,
3282
+ "loss": 1.1715,
3283
+ "step": 455
3284
+ },
3285
+ {
3286
+ "epoch": 1.2705718270571826,
3287
+ "grad_norm": 1.019865022262857,
3288
+ "learning_rate": 6.497442523029663e-06,
3289
+ "loss": 1.3491,
3290
+ "step": 456
3291
+ },
3292
+ {
3293
+ "epoch": 1.2733612273361228,
3294
+ "grad_norm": 1.1857253940298278,
3295
+ "learning_rate": 6.453951129574644e-06,
3296
+ "loss": 1.2559,
3297
+ "step": 457
3298
+ },
3299
+ {
3300
+ "epoch": 1.2761506276150627,
3301
+ "grad_norm": 1.082634058798414,
3302
+ "learning_rate": 6.410536322278523e-06,
3303
+ "loss": 1.1154,
3304
+ "step": 458
3305
+ },
3306
+ {
3307
+ "epoch": 1.2789400278940029,
3308
+ "grad_norm": 1.212134850206044,
3309
+ "learning_rate": 6.3671990387971096e-06,
3310
+ "loss": 1.3202,
3311
+ "step": 459
3312
+ },
3313
+ {
3314
+ "epoch": 1.2817294281729428,
3315
+ "grad_norm": 1.0917010213427312,
3316
+ "learning_rate": 6.32394021511189e-06,
3317
+ "loss": 1.3484,
3318
+ "step": 460
3319
+ },
3320
+ {
3321
+ "epoch": 1.284518828451883,
3322
+ "grad_norm": 1.1428008219567098,
3323
+ "learning_rate": 6.280760785509802e-06,
3324
+ "loss": 1.1919,
3325
+ "step": 461
3326
+ },
3327
+ {
3328
+ "epoch": 1.2873082287308228,
3329
+ "grad_norm": 1.0825996728928653,
3330
+ "learning_rate": 6.237661682563055e-06,
3331
+ "loss": 1.2082,
3332
+ "step": 462
3333
+ },
3334
+ {
3335
+ "epoch": 1.2900976290097628,
3336
+ "grad_norm": 1.1848809666798936,
3337
+ "learning_rate": 6.194643837109015e-06,
3338
+ "loss": 1.2828,
3339
+ "step": 463
3340
+ },
3341
+ {
3342
+ "epoch": 1.292887029288703,
3343
+ "grad_norm": 1.0895688626081312,
3344
+ "learning_rate": 6.151708178230057e-06,
3345
+ "loss": 1.2387,
3346
+ "step": 464
3347
+ },
3348
+ {
3349
+ "epoch": 1.295676429567643,
3350
+ "grad_norm": 1.1868518830616108,
3351
+ "learning_rate": 6.108855633233546e-06,
3352
+ "loss": 1.297,
3353
+ "step": 465
3354
+ },
3355
+ {
3356
+ "epoch": 1.298465829846583,
3357
+ "grad_norm": 1.1228238807592477,
3358
+ "learning_rate": 6.066087127631761e-06,
3359
+ "loss": 1.4407,
3360
+ "step": 466
3361
+ },
3362
+ {
3363
+ "epoch": 1.301255230125523,
3364
+ "grad_norm": 1.17982652645882,
3365
+ "learning_rate": 6.0234035851219604e-06,
3366
+ "loss": 1.2222,
3367
+ "step": 467
3368
+ },
3369
+ {
3370
+ "epoch": 1.304044630404463,
3371
+ "grad_norm": 1.2059016405123564,
3372
+ "learning_rate": 5.9808059275663775e-06,
3373
+ "loss": 1.2743,
3374
+ "step": 468
3375
+ },
3376
+ {
3377
+ "epoch": 1.3068340306834032,
3378
+ "grad_norm": 1.2223828433454567,
3379
+ "learning_rate": 5.93829507497235e-06,
3380
+ "loss": 1.4048,
3381
+ "step": 469
3382
+ },
3383
+ {
3384
+ "epoch": 1.3096234309623431,
3385
+ "grad_norm": 1.2327820107615224,
3386
+ "learning_rate": 5.895871945472434e-06,
3387
+ "loss": 1.5033,
3388
+ "step": 470
3389
+ },
3390
+ {
3391
+ "epoch": 1.312412831241283,
3392
+ "grad_norm": 1.1838495789031667,
3393
+ "learning_rate": 5.853537455304575e-06,
3394
+ "loss": 1.2618,
3395
+ "step": 471
3396
+ },
3397
+ {
3398
+ "epoch": 1.3152022315202232,
3399
+ "grad_norm": 1.0841644055995976,
3400
+ "learning_rate": 5.811292518792324e-06,
3401
+ "loss": 1.453,
3402
+ "step": 472
3403
+ },
3404
+ {
3405
+ "epoch": 1.3179916317991631,
3406
+ "grad_norm": 1.3004967203117666,
3407
+ "learning_rate": 5.769138048325087e-06,
3408
+ "loss": 1.2257,
3409
+ "step": 473
3410
+ },
3411
+ {
3412
+ "epoch": 1.3207810320781033,
3413
+ "grad_norm": 1.1043354920526318,
3414
+ "learning_rate": 5.72707495433842e-06,
3415
+ "loss": 1.2821,
3416
+ "step": 474
3417
+ },
3418
+ {
3419
+ "epoch": 1.3235704323570432,
3420
+ "grad_norm": 1.2877538618327764,
3421
+ "learning_rate": 5.685104145294364e-06,
3422
+ "loss": 1.214,
3423
+ "step": 475
3424
+ },
3425
+ {
3426
+ "epoch": 1.3263598326359833,
3427
+ "grad_norm": 1.1788804153888472,
3428
+ "learning_rate": 5.643226527661825e-06,
3429
+ "loss": 1.3006,
3430
+ "step": 476
3431
+ },
3432
+ {
3433
+ "epoch": 1.3291492329149233,
3434
+ "grad_norm": 1.0526011395627175,
3435
+ "learning_rate": 5.601443005897012e-06,
3436
+ "loss": 1.3178,
3437
+ "step": 477
3438
+ },
3439
+ {
3440
+ "epoch": 1.3319386331938632,
3441
+ "grad_norm": 1.0726595908035388,
3442
+ "learning_rate": 5.55975448242386e-06,
3443
+ "loss": 1.3458,
3444
+ "step": 478
3445
+ },
3446
+ {
3447
+ "epoch": 1.3347280334728033,
3448
+ "grad_norm": 1.0976574017960397,
3449
+ "learning_rate": 5.5181618576146e-06,
3450
+ "loss": 1.3022,
3451
+ "step": 479
3452
+ },
3453
+ {
3454
+ "epoch": 1.3375174337517435,
3455
+ "grad_norm": 1.2225185516827237,
3456
+ "learning_rate": 5.476666029770259e-06,
3457
+ "loss": 1.3442,
3458
+ "step": 480
3459
+ },
3460
+ {
3461
+ "epoch": 1.3403068340306834,
3462
+ "grad_norm": 1.0680703822354123,
3463
+ "learning_rate": 5.435267895101303e-06,
3464
+ "loss": 1.2985,
3465
+ "step": 481
3466
+ },
3467
+ {
3468
+ "epoch": 1.3430962343096233,
3469
+ "grad_norm": 1.186589064765597,
3470
+ "learning_rate": 5.393968347708233e-06,
3471
+ "loss": 1.1773,
3472
+ "step": 482
3473
+ },
3474
+ {
3475
+ "epoch": 1.3458856345885635,
3476
+ "grad_norm": 1.0143150489355277,
3477
+ "learning_rate": 5.352768279562315e-06,
3478
+ "loss": 1.313,
3479
+ "step": 483
3480
+ },
3481
+ {
3482
+ "epoch": 1.3486750348675036,
3483
+ "grad_norm": 1.2351862249444747,
3484
+ "learning_rate": 5.311668580486311e-06,
3485
+ "loss": 1.3821,
3486
+ "step": 484
3487
+ },
3488
+ {
3489
+ "epoch": 1.3514644351464435,
3490
+ "grad_norm": 1.1346210320256067,
3491
+ "learning_rate": 5.270670138135234e-06,
3492
+ "loss": 1.334,
3493
+ "step": 485
3494
+ },
3495
+ {
3496
+ "epoch": 1.3542538354253835,
3497
+ "grad_norm": 1.2118710215904622,
3498
+ "learning_rate": 5.229773837977208e-06,
3499
+ "loss": 1.3088,
3500
+ "step": 486
3501
+ },
3502
+ {
3503
+ "epoch": 1.3570432357043236,
3504
+ "grad_norm": 1.0342577775352215,
3505
+ "learning_rate": 5.188980563274315e-06,
3506
+ "loss": 1.1462,
3507
+ "step": 487
3508
+ },
3509
+ {
3510
+ "epoch": 1.3598326359832635,
3511
+ "grad_norm": 1.0728116890130244,
3512
+ "learning_rate": 5.148291195063555e-06,
3513
+ "loss": 1.2636,
3514
+ "step": 488
3515
+ },
3516
+ {
3517
+ "epoch": 1.3626220362622037,
3518
+ "grad_norm": 1.1893323626916112,
3519
+ "learning_rate": 5.107706612137776e-06,
3520
+ "loss": 1.2052,
3521
+ "step": 489
3522
+ },
3523
+ {
3524
+ "epoch": 1.3654114365411436,
3525
+ "grad_norm": 1.0157712327690198,
3526
+ "learning_rate": 5.067227691026737e-06,
3527
+ "loss": 1.1938,
3528
+ "step": 490
3529
+ },
3530
+ {
3531
+ "epoch": 1.3682008368200838,
3532
+ "grad_norm": 1.2326467093213043,
3533
+ "learning_rate": 5.026855305978129e-06,
3534
+ "loss": 1.2607,
3535
+ "step": 491
3536
+ },
3537
+ {
3538
+ "epoch": 1.3709902370990237,
3539
+ "grad_norm": 1.049878301655513,
3540
+ "learning_rate": 4.986590328938734e-06,
3541
+ "loss": 1.3024,
3542
+ "step": 492
3543
+ },
3544
+ {
3545
+ "epoch": 1.3737796373779636,
3546
+ "grad_norm": 1.0826344733380577,
3547
+ "learning_rate": 4.946433629535585e-06,
3548
+ "loss": 1.2758,
3549
+ "step": 493
3550
+ },
3551
+ {
3552
+ "epoch": 1.3765690376569037,
3553
+ "grad_norm": 1.1061892981697474,
3554
+ "learning_rate": 4.906386075057164e-06,
3555
+ "loss": 1.1803,
3556
+ "step": 494
3557
+ },
3558
+ {
3559
+ "epoch": 1.379358437935844,
3560
+ "grad_norm": 1.08675011661269,
3561
+ "learning_rate": 4.866448530434692e-06,
3562
+ "loss": 1.2746,
3563
+ "step": 495
3564
+ },
3565
+ {
3566
+ "epoch": 1.379358437935844,
3567
+ "eval_loss": 1.573391318321228,
3568
+ "eval_runtime": 157.8173,
3569
+ "eval_samples_per_second": 2.015,
3570
+ "eval_steps_per_second": 0.127,
3571
+ "step": 495
3572
+ },
3573
+ {
3574
+ "epoch": 1.3821478382147838,
3575
+ "grad_norm": 1.1281238857498397,
3576
+ "learning_rate": 4.826621858223431e-06,
3577
+ "loss": 1.357,
3578
+ "step": 496
3579
+ },
3580
+ {
3581
+ "epoch": 1.3849372384937237,
3582
+ "grad_norm": 1.055676305380402,
3583
+ "learning_rate": 4.786906918584083e-06,
3584
+ "loss": 1.2167,
3585
+ "step": 497
3586
+ },
3587
+ {
3588
+ "epoch": 1.387726638772664,
3589
+ "grad_norm": 1.213376805413726,
3590
+ "learning_rate": 4.747304569264179e-06,
3591
+ "loss": 1.4123,
3592
+ "step": 498
3593
+ },
3594
+ {
3595
+ "epoch": 1.390516039051604,
3596
+ "grad_norm": 1.0574410266065954,
3597
+ "learning_rate": 4.707815665579569e-06,
3598
+ "loss": 1.3955,
3599
+ "step": 499
3600
+ },
3601
+ {
3602
+ "epoch": 1.393305439330544,
3603
+ "grad_norm": 1.2166871008897608,
3604
+ "learning_rate": 4.668441060395958e-06,
3605
+ "loss": 1.3053,
3606
+ "step": 500
3607
+ },
3608
+ {
3609
+ "epoch": 1.3960948396094839,
3610
+ "grad_norm": 1.0626532706218652,
3611
+ "learning_rate": 4.629181604110464e-06,
3612
+ "loss": 1.2975,
3613
+ "step": 501
3614
+ },
3615
+ {
3616
+ "epoch": 1.398884239888424,
3617
+ "grad_norm": 1.2021150374539924,
3618
+ "learning_rate": 4.5900381446332854e-06,
3619
+ "loss": 1.2329,
3620
+ "step": 502
3621
+ },
3622
+ {
3623
+ "epoch": 1.401673640167364,
3624
+ "grad_norm": 1.121645372017073,
3625
+ "learning_rate": 4.551011527369348e-06,
3626
+ "loss": 1.2224,
3627
+ "step": 503
3628
+ },
3629
+ {
3630
+ "epoch": 1.404463040446304,
3631
+ "grad_norm": 1.1242323642605798,
3632
+ "learning_rate": 4.512102595200073e-06,
3633
+ "loss": 1.2218,
3634
+ "step": 504
3635
+ },
3636
+ {
3637
+ "epoch": 1.407252440725244,
3638
+ "grad_norm": 1.1250999334354057,
3639
+ "learning_rate": 4.4733121884651665e-06,
3640
+ "loss": 1.3524,
3641
+ "step": 505
3642
+ },
3643
+ {
3644
+ "epoch": 1.4100418410041842,
3645
+ "grad_norm": 1.190748096280525,
3646
+ "learning_rate": 4.434641144944464e-06,
3647
+ "loss": 1.2794,
3648
+ "step": 506
3649
+ },
3650
+ {
3651
+ "epoch": 1.412831241283124,
3652
+ "grad_norm": 1.1256136466982283,
3653
+ "learning_rate": 4.3960902998398524e-06,
3654
+ "loss": 1.2871,
3655
+ "step": 507
3656
+ },
3657
+ {
3658
+ "epoch": 1.415620641562064,
3659
+ "grad_norm": 1.3231624966545141,
3660
+ "learning_rate": 4.357660485757211e-06,
3661
+ "loss": 1.3109,
3662
+ "step": 508
3663
+ },
3664
+ {
3665
+ "epoch": 1.4184100418410042,
3666
+ "grad_norm": 1.1507561161367288,
3667
+ "learning_rate": 4.319352532688444e-06,
3668
+ "loss": 1.2732,
3669
+ "step": 509
3670
+ },
3671
+ {
3672
+ "epoch": 1.4211994421199443,
3673
+ "grad_norm": 1.2022076233431473,
3674
+ "learning_rate": 4.281167267993541e-06,
3675
+ "loss": 1.2193,
3676
+ "step": 510
3677
+ },
3678
+ {
3679
+ "epoch": 1.4239888423988842,
3680
+ "grad_norm": 1.1014114088374245,
3681
+ "learning_rate": 4.243105516382732e-06,
3682
+ "loss": 1.2282,
3683
+ "step": 511
3684
+ },
3685
+ {
3686
+ "epoch": 1.4267782426778242,
3687
+ "grad_norm": 1.1851428019160586,
3688
+ "learning_rate": 4.205168099898652e-06,
3689
+ "loss": 1.3455,
3690
+ "step": 512
3691
+ },
3692
+ {
3693
+ "epoch": 1.4295676429567643,
3694
+ "grad_norm": 1.262340605187208,
3695
+ "learning_rate": 4.167355837898585e-06,
3696
+ "loss": 1.3841,
3697
+ "step": 513
3698
+ },
3699
+ {
3700
+ "epoch": 1.4323570432357045,
3701
+ "grad_norm": 1.1995371048652899,
3702
+ "learning_rate": 4.129669547036798e-06,
3703
+ "loss": 1.3889,
3704
+ "step": 514
3705
+ },
3706
+ {
3707
+ "epoch": 1.4351464435146444,
3708
+ "grad_norm": 1.1534627899877103,
3709
+ "learning_rate": 4.092110041246865e-06,
3710
+ "loss": 1.323,
3711
+ "step": 515
3712
+ },
3713
+ {
3714
+ "epoch": 1.4379358437935843,
3715
+ "grad_norm": 1.0717749801066352,
3716
+ "learning_rate": 4.054678131724128e-06,
3717
+ "loss": 1.1756,
3718
+ "step": 516
3719
+ },
3720
+ {
3721
+ "epoch": 1.4407252440725244,
3722
+ "grad_norm": 1.0332565205169049,
3723
+ "learning_rate": 4.017374626908125e-06,
3724
+ "loss": 1.2921,
3725
+ "step": 517
3726
+ },
3727
+ {
3728
+ "epoch": 1.4435146443514644,
3729
+ "grad_norm": 1.299221266861857,
3730
+ "learning_rate": 3.980200332465189e-06,
3731
+ "loss": 1.503,
3732
+ "step": 518
3733
+ },
3734
+ {
3735
+ "epoch": 1.4463040446304045,
3736
+ "grad_norm": 1.1258042918602684,
3737
+ "learning_rate": 3.943156051271003e-06,
3738
+ "loss": 1.326,
3739
+ "step": 519
3740
+ },
3741
+ {
3742
+ "epoch": 1.4490934449093444,
3743
+ "grad_norm": 1.1498347580379522,
3744
+ "learning_rate": 3.906242583393279e-06,
3745
+ "loss": 1.2696,
3746
+ "step": 520
3747
+ },
3748
+ {
3749
+ "epoch": 1.4518828451882846,
3750
+ "grad_norm": 1.0518934141037535,
3751
+ "learning_rate": 3.8694607260744745e-06,
3752
+ "loss": 1.2943,
3753
+ "step": 521
3754
+ },
3755
+ {
3756
+ "epoch": 1.4546722454672245,
3757
+ "grad_norm": 1.0909311949978682,
3758
+ "learning_rate": 3.832811273714569e-06,
3759
+ "loss": 1.2605,
3760
+ "step": 522
3761
+ },
3762
+ {
3763
+ "epoch": 1.4574616457461644,
3764
+ "grad_norm": 1.085659967414048,
3765
+ "learning_rate": 3.7962950178539282e-06,
3766
+ "loss": 1.162,
3767
+ "step": 523
3768
+ },
3769
+ {
3770
+ "epoch": 1.4602510460251046,
3771
+ "grad_norm": 1.0373035764107983,
3772
+ "learning_rate": 3.7599127471561746e-06,
3773
+ "loss": 1.2543,
3774
+ "step": 524
3775
+ },
3776
+ {
3777
+ "epoch": 1.4630404463040447,
3778
+ "grad_norm": 1.195186829033534,
3779
+ "learning_rate": 3.7236652473911817e-06,
3780
+ "loss": 1.1093,
3781
+ "step": 525
3782
+ },
3783
+ {
3784
+ "epoch": 1.4658298465829847,
3785
+ "grad_norm": 1.060512460431333,
3786
+ "learning_rate": 3.687553301418092e-06,
3787
+ "loss": 1.1976,
3788
+ "step": 526
3789
+ },
3790
+ {
3791
+ "epoch": 1.4686192468619246,
3792
+ "grad_norm": 1.0036394866362008,
3793
+ "learning_rate": 3.651577689168405e-06,
3794
+ "loss": 1.3004,
3795
+ "step": 527
3796
+ },
3797
+ {
3798
+ "epoch": 1.4714086471408647,
3799
+ "grad_norm": 1.113980704434126,
3800
+ "learning_rate": 3.6157391876291535e-06,
3801
+ "loss": 1.2108,
3802
+ "step": 528
3803
+ },
3804
+ {
3805
+ "epoch": 1.4741980474198049,
3806
+ "grad_norm": 1.0528377393119437,
3807
+ "learning_rate": 3.580038570826093e-06,
3808
+ "loss": 1.1086,
3809
+ "step": 529
3810
+ },
3811
+ {
3812
+ "epoch": 1.4769874476987448,
3813
+ "grad_norm": 1.1766676130844111,
3814
+ "learning_rate": 3.5444766098070037e-06,
3815
+ "loss": 1.4426,
3816
+ "step": 530
3817
+ },
3818
+ {
3819
+ "epoch": 1.4797768479776847,
3820
+ "grad_norm": 1.2618504608225378,
3821
+ "learning_rate": 3.509054072625031e-06,
3822
+ "loss": 1.3814,
3823
+ "step": 531
3824
+ },
3825
+ {
3826
+ "epoch": 1.4825662482566249,
3827
+ "grad_norm": 1.2891981764585003,
3828
+ "learning_rate": 3.473771724322108e-06,
3829
+ "loss": 1.2466,
3830
+ "step": 532
3831
+ },
3832
+ {
3833
+ "epoch": 1.4853556485355648,
3834
+ "grad_norm": 1.0110218638691248,
3835
+ "learning_rate": 3.4386303269124142e-06,
3836
+ "loss": 1.1999,
3837
+ "step": 533
3838
+ },
3839
+ {
3840
+ "epoch": 1.488145048814505,
3841
+ "grad_norm": 1.2034400983007125,
3842
+ "learning_rate": 3.4036306393659324e-06,
3843
+ "loss": 1.3068,
3844
+ "step": 534
3845
+ },
3846
+ {
3847
+ "epoch": 1.4909344490934449,
3848
+ "grad_norm": 1.1385421777709188,
3849
+ "learning_rate": 3.3687734175920505e-06,
3850
+ "loss": 1.1693,
3851
+ "step": 535
3852
+ },
3853
+ {
3854
+ "epoch": 1.493723849372385,
3855
+ "grad_norm": 1.1564188245202627,
3856
+ "learning_rate": 3.334059414423233e-06,
3857
+ "loss": 1.2479,
3858
+ "step": 536
3859
+ },
3860
+ {
3861
+ "epoch": 1.496513249651325,
3862
+ "grad_norm": 1.0632144492289124,
3863
+ "learning_rate": 3.299489379598777e-06,
3864
+ "loss": 1.2818,
3865
+ "step": 537
3866
+ }
3867
+ ],
3868
+ "logging_steps": 1,
3869
+ "max_steps": 716,
3870
+ "num_input_tokens_seen": 0,
3871
+ "num_train_epochs": 2,
3872
+ "save_steps": 179,
3873
+ "stateful_callbacks": {
3874
+ "TrainerControl": {
3875
+ "args": {
3876
+ "should_epoch_stop": false,
3877
+ "should_evaluate": false,
3878
+ "should_log": false,
3879
+ "should_save": true,
3880
+ "should_training_stop": false
3881
+ },
3882
+ "attributes": {}
3883
+ }
3884
+ },
3885
+ "total_flos": 349631812730880.0,
3886
+ "train_batch_size": 4,
3887
+ "trial_name": null,
3888
+ "trial_params": null
3889
+ }
checkpoint-537/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:176779fd314d79db1323a9603622ba6af7ce08e6291ea5cb8066c39938973900
3
+ size 8504
checkpoint-537/zero_to_fp32.py ADDED
@@ -0,0 +1,674 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python
2
+
3
+ # Copyright (c) Microsoft Corporation.
4
+ # SPDX-License-Identifier: Apache-2.0
5
+
6
+ # DeepSpeed Team
7
+
8
+ # This script extracts fp32 consolidated weights from a zero 1, 2 and 3 DeepSpeed checkpoints. It gets
9
+ # copied into the top level checkpoint dir, so the user can easily do the conversion at any point in
10
+ # the future. Once extracted, the weights don't require DeepSpeed and can be used in any
11
+ # application.
12
+ #
13
+ # example:
14
+ # python zero_to_fp32.py . output_dir/
15
+ # or
16
+ # python zero_to_fp32.py . output_dir/ --safe_serialization
17
+
18
+ import argparse
19
+ import torch
20
+ import glob
21
+ import math
22
+ import os
23
+ import re
24
+ import json
25
+ from tqdm import tqdm
26
+ from collections import OrderedDict
27
+ from dataclasses import dataclass
28
+
29
+ # while this script doesn't use deepspeed to recover data, since the checkpoints are pickled with
30
+ # DeepSpeed data structures it has to be available in the current python environment.
31
+ from deepspeed.utils import logger
32
+ from deepspeed.checkpoint.constants import (DS_VERSION, OPTIMIZER_STATE_DICT, SINGLE_PARTITION_OF_FP32_GROUPS,
33
+ FP32_FLAT_GROUPS, ZERO_STAGE, PARTITION_COUNT, PARAM_SHAPES, BUFFER_NAMES,
34
+ FROZEN_PARAM_SHAPES, FROZEN_PARAM_FRAGMENTS)
35
+
36
+
37
+ @dataclass
38
+ class zero_model_state:
39
+ buffers: dict()
40
+ param_shapes: dict()
41
+ shared_params: list
42
+ ds_version: int
43
+ frozen_param_shapes: dict()
44
+ frozen_param_fragments: dict()
45
+
46
+
47
+ debug = 0
48
+
49
+ # load to cpu
50
+ device = torch.device('cpu')
51
+
52
+
53
+ def atoi(text):
54
+ return int(text) if text.isdigit() else text
55
+
56
+
57
+ def natural_keys(text):
58
+ '''
59
+ alist.sort(key=natural_keys) sorts in human order
60
+ http://nedbatchelder.com/blog/200712/human_sorting.html
61
+ (See Toothy's implementation in the comments)
62
+ '''
63
+ return [atoi(c) for c in re.split(r'(\d+)', text)]
64
+
65
+
66
+ def get_model_state_file(checkpoint_dir, zero_stage):
67
+ if not os.path.isdir(checkpoint_dir):
68
+ raise FileNotFoundError(f"Directory '{checkpoint_dir}' doesn't exist")
69
+
70
+ # there should be only one file
71
+ if zero_stage <= 2:
72
+ file = os.path.join(checkpoint_dir, "mp_rank_00_model_states.pt")
73
+ elif zero_stage == 3:
74
+ file = os.path.join(checkpoint_dir, "zero_pp_rank_0_mp_rank_00_model_states.pt")
75
+
76
+ if not os.path.exists(file):
77
+ raise FileNotFoundError(f"can't find model states file at '{file}'")
78
+
79
+ return file
80
+
81
+
82
+ def get_checkpoint_files(checkpoint_dir, glob_pattern):
83
+ # XXX: need to test that this simple glob rule works for multi-node setup too
84
+ ckpt_files = sorted(glob.glob(os.path.join(checkpoint_dir, glob_pattern)), key=natural_keys)
85
+
86
+ if len(ckpt_files) == 0:
87
+ raise FileNotFoundError(f"can't find {glob_pattern} files in directory '{checkpoint_dir}'")
88
+
89
+ return ckpt_files
90
+
91
+
92
+ def get_optim_files(checkpoint_dir):
93
+ return get_checkpoint_files(checkpoint_dir, "*_optim_states.pt")
94
+
95
+
96
+ def get_model_state_files(checkpoint_dir):
97
+ return get_checkpoint_files(checkpoint_dir, "*_model_states.pt")
98
+
99
+
100
+ def parse_model_states(files):
101
+ zero_model_states = []
102
+ for file in files:
103
+ state_dict = torch.load(file, map_location=device)
104
+
105
+ if BUFFER_NAMES not in state_dict:
106
+ raise ValueError(f"{file} is not a model state checkpoint")
107
+ buffer_names = state_dict[BUFFER_NAMES]
108
+ if debug:
109
+ print("Found buffers:", buffer_names)
110
+
111
+ # recover just the buffers while restoring them to fp32 if they were saved in fp16
112
+ buffers = {k: v.float() for k, v in state_dict["module"].items() if k in buffer_names}
113
+ param_shapes = state_dict[PARAM_SHAPES]
114
+
115
+ # collect parameters that are included in param_shapes
116
+ param_names = []
117
+ for s in param_shapes:
118
+ for name in s.keys():
119
+ param_names.append(name)
120
+
121
+ # update with frozen parameters
122
+ frozen_param_shapes = state_dict.get(FROZEN_PARAM_SHAPES, None)
123
+ if frozen_param_shapes is not None:
124
+ if debug:
125
+ print(f"Found frozen_param_shapes: {frozen_param_shapes}")
126
+ param_names += list(frozen_param_shapes.keys())
127
+
128
+ # handle shared params
129
+ shared_params = [[k, v] for k, v in state_dict["shared_params"].items()]
130
+
131
+ ds_version = state_dict.get(DS_VERSION, None)
132
+
133
+ frozen_param_fragments = state_dict.get(FROZEN_PARAM_FRAGMENTS, None)
134
+
135
+ z_model_state = zero_model_state(buffers=buffers,
136
+ param_shapes=param_shapes,
137
+ shared_params=shared_params,
138
+ ds_version=ds_version,
139
+ frozen_param_shapes=frozen_param_shapes,
140
+ frozen_param_fragments=frozen_param_fragments)
141
+ zero_model_states.append(z_model_state)
142
+
143
+ return zero_model_states
144
+
145
+
146
+ def parse_optim_states(files, ds_checkpoint_dir):
147
+ total_files = len(files)
148
+ state_dicts = []
149
+ for f in files:
150
+ state_dict = torch.load(f, map_location=device)
151
+ # immediately discard the potentially huge 2 optimizer states as we only care for fp32 master weights
152
+ # and also handle the case where it was already removed by another helper script
153
+ state_dict["optimizer_state_dict"].pop("optimizer_state_dict", None)
154
+ state_dicts.append(state_dict)
155
+
156
+ if not ZERO_STAGE in state_dicts[0][OPTIMIZER_STATE_DICT]:
157
+ raise ValueError(f"{files[0]} is not a zero checkpoint")
158
+ zero_stage = state_dicts[0][OPTIMIZER_STATE_DICT][ZERO_STAGE]
159
+ world_size = state_dicts[0][OPTIMIZER_STATE_DICT][PARTITION_COUNT]
160
+
161
+ # For ZeRO-2 each param group can have different partition_count as data parallelism for expert
162
+ # parameters can be different from data parallelism for non-expert parameters. So we can just
163
+ # use the max of the partition_count to get the dp world_size.
164
+
165
+ if type(world_size) is list:
166
+ world_size = max(world_size)
167
+
168
+ if world_size != total_files:
169
+ raise ValueError(
170
+ f"Expected {world_size} of '*_optim_states.pt' under '{ds_checkpoint_dir}' but found {total_files} files. "
171
+ "Possibly due to an overwrite of an old checkpoint, or a checkpoint didn't get saved by one or more processes."
172
+ )
173
+
174
+ # the groups are named differently in each stage
175
+ if zero_stage <= 2:
176
+ fp32_groups_key = SINGLE_PARTITION_OF_FP32_GROUPS
177
+ elif zero_stage == 3:
178
+ fp32_groups_key = FP32_FLAT_GROUPS
179
+ else:
180
+ raise ValueError(f"unknown zero stage {zero_stage}")
181
+
182
+ if zero_stage <= 2:
183
+ fp32_flat_groups = [state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key] for i in range(len(state_dicts))]
184
+ elif zero_stage == 3:
185
+ # if there is more than one param group, there will be multiple flattened tensors - one
186
+ # flattened tensor per group - for simplicity merge them into a single tensor
187
+ #
188
+ # XXX: could make the script more memory efficient for when there are multiple groups - it
189
+ # will require matching the sub-lists of param_shapes for each param group flattened tensor
190
+
191
+ fp32_flat_groups = [
192
+ torch.cat(state_dicts[i][OPTIMIZER_STATE_DICT][fp32_groups_key], 0) for i in range(len(state_dicts))
193
+ ]
194
+
195
+ return zero_stage, world_size, fp32_flat_groups
196
+
197
+
198
+ def _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters):
199
+ """
200
+ Returns fp32 state_dict reconstructed from ds checkpoint
201
+
202
+ Args:
203
+ - ``ds_checkpoint_dir``: path to the deepspeed checkpoint folder (where the optimizer files are)
204
+
205
+ """
206
+ print(f"Processing zero checkpoint '{ds_checkpoint_dir}'")
207
+
208
+ optim_files = get_optim_files(ds_checkpoint_dir)
209
+ zero_stage, world_size, fp32_flat_groups = parse_optim_states(optim_files, ds_checkpoint_dir)
210
+ print(f"Detected checkpoint of type zero stage {zero_stage}, world_size: {world_size}")
211
+
212
+ model_files = get_model_state_files(ds_checkpoint_dir)
213
+
214
+ zero_model_states = parse_model_states(model_files)
215
+ print(f'Parsing checkpoint created by deepspeed=={zero_model_states[0].ds_version}')
216
+
217
+ if zero_stage <= 2:
218
+ return _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
219
+ exclude_frozen_parameters)
220
+ elif zero_stage == 3:
221
+ return _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
222
+ exclude_frozen_parameters)
223
+
224
+
225
+ def _zero2_merge_frozen_params(state_dict, zero_model_states):
226
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
227
+ return
228
+
229
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
230
+ frozen_param_fragments = zero_model_states[0].frozen_param_fragments
231
+
232
+ if debug:
233
+ num_elem = sum(s.numel() for s in frozen_param_shapes.values())
234
+ print(f'rank 0: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
235
+
236
+ wanted_params = len(frozen_param_shapes)
237
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
238
+ avail_numel = sum([p.numel() for p in frozen_param_fragments.values()])
239
+ print(f'Frozen params: Have {avail_numel} numels to process.')
240
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
241
+
242
+ total_params = 0
243
+ total_numel = 0
244
+ for name, shape in frozen_param_shapes.items():
245
+ total_params += 1
246
+ unpartitioned_numel = shape.numel()
247
+ total_numel += unpartitioned_numel
248
+
249
+ state_dict[name] = frozen_param_fragments[name]
250
+
251
+ if debug:
252
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
253
+
254
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
255
+
256
+
257
+ def _has_callable(obj, fn):
258
+ attr = getattr(obj, fn, None)
259
+ return callable(attr)
260
+
261
+
262
+ def _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
263
+ param_shapes = zero_model_states[0].param_shapes
264
+
265
+ # Reconstruction protocol:
266
+ #
267
+ # XXX: document this
268
+
269
+ if debug:
270
+ for i in range(world_size):
271
+ for j in range(len(fp32_flat_groups[0])):
272
+ print(f"{FP32_FLAT_GROUPS}[{i}][{j}].shape={fp32_flat_groups[i][j].shape}")
273
+
274
+ # XXX: memory usage doubles here (zero2)
275
+ num_param_groups = len(fp32_flat_groups[0])
276
+ merged_single_partition_of_fp32_groups = []
277
+ for i in range(num_param_groups):
278
+ merged_partitions = [sd[i] for sd in fp32_flat_groups]
279
+ full_single_fp32_vector = torch.cat(merged_partitions, 0)
280
+ merged_single_partition_of_fp32_groups.append(full_single_fp32_vector)
281
+ avail_numel = sum(
282
+ [full_single_fp32_vector.numel() for full_single_fp32_vector in merged_single_partition_of_fp32_groups])
283
+
284
+ if debug:
285
+ wanted_params = sum([len(shapes) for shapes in param_shapes])
286
+ wanted_numel = sum([sum(shape.numel() for shape in shapes.values()) for shapes in param_shapes])
287
+ # not asserting if there is a mismatch due to possible padding
288
+ print(f"Have {avail_numel} numels to process.")
289
+ print(f"Need {wanted_numel} numels in {wanted_params} params.")
290
+
291
+ # params
292
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
293
+ # out-of-core computing solution
294
+ total_numel = 0
295
+ total_params = 0
296
+ for shapes, full_single_fp32_vector in zip(param_shapes, merged_single_partition_of_fp32_groups):
297
+ offset = 0
298
+ avail_numel = full_single_fp32_vector.numel()
299
+ for name, shape in shapes.items():
300
+
301
+ unpartitioned_numel = shape.numel() if _has_callable(shape, 'numel') else math.prod(shape)
302
+ total_numel += unpartitioned_numel
303
+ total_params += 1
304
+
305
+ if debug:
306
+ print(f"{name} full shape: {shape} unpartitioned numel {unpartitioned_numel} ")
307
+ state_dict[name] = full_single_fp32_vector.narrow(0, offset, unpartitioned_numel).view(shape)
308
+ offset += unpartitioned_numel
309
+
310
+ # Z2 started to align to 2*world_size to improve nccl performance. Therefore both offset and
311
+ # avail_numel can differ by anywhere between 0..2*world_size. Due to two unrelated complex
312
+ # paddings performed in the code it's almost impossible to predict the exact numbers w/o the
313
+ # live optimizer object, so we are checking that the numbers are within the right range
314
+ align_to = 2 * world_size
315
+
316
+ def zero2_align(x):
317
+ return align_to * math.ceil(x / align_to)
318
+
319
+ if debug:
320
+ print(f"original offset={offset}, avail_numel={avail_numel}")
321
+
322
+ offset = zero2_align(offset)
323
+ avail_numel = zero2_align(avail_numel)
324
+
325
+ if debug:
326
+ print(f"aligned offset={offset}, avail_numel={avail_numel}")
327
+
328
+ # Sanity check
329
+ if offset != avail_numel:
330
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
331
+
332
+ print(f"Reconstructed fp32 state dict with {total_params} params {total_numel} elements")
333
+
334
+
335
+ def _get_fp32_state_dict_from_zero2_checkpoint(world_size, fp32_flat_groups, zero_model_states,
336
+ exclude_frozen_parameters):
337
+ state_dict = OrderedDict()
338
+
339
+ # buffers
340
+ buffers = zero_model_states[0].buffers
341
+ state_dict.update(buffers)
342
+ if debug:
343
+ print(f"added {len(buffers)} buffers")
344
+
345
+ if not exclude_frozen_parameters:
346
+ _zero2_merge_frozen_params(state_dict, zero_model_states)
347
+
348
+ _zero2_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
349
+
350
+ # recover shared parameters
351
+ for pair in zero_model_states[0].shared_params:
352
+ if pair[1] in state_dict:
353
+ state_dict[pair[0]] = state_dict[pair[1]]
354
+
355
+ return state_dict
356
+
357
+
358
+ def zero3_partitioned_param_info(unpartitioned_numel, world_size):
359
+ remainder = unpartitioned_numel % world_size
360
+ padding_numel = (world_size - remainder) if remainder else 0
361
+ partitioned_numel = math.ceil(unpartitioned_numel / world_size)
362
+ return partitioned_numel, padding_numel
363
+
364
+
365
+ def _zero3_merge_frozen_params(state_dict, world_size, zero_model_states):
366
+ if zero_model_states[0].frozen_param_shapes is None or len(zero_model_states[0].frozen_param_shapes) == 0:
367
+ return
368
+
369
+ if debug:
370
+ for i in range(world_size):
371
+ num_elem = sum(s.numel() for s in zero_model_states[i].frozen_param_fragments.values())
372
+ print(f'rank {i}: {FROZEN_PARAM_SHAPES}.numel = {num_elem}')
373
+
374
+ frozen_param_shapes = zero_model_states[0].frozen_param_shapes
375
+ wanted_params = len(frozen_param_shapes)
376
+ wanted_numel = sum(s.numel() for s in frozen_param_shapes.values())
377
+ avail_numel = sum([p.numel() for p in zero_model_states[0].frozen_param_fragments.values()]) * world_size
378
+ print(f'Frozen params: Have {avail_numel} numels to process.')
379
+ print(f'Frozen params: Need {wanted_numel} numels in {wanted_params} params')
380
+
381
+ total_params = 0
382
+ total_numel = 0
383
+ for name, shape in zero_model_states[0].frozen_param_shapes.items():
384
+ total_params += 1
385
+ unpartitioned_numel = shape.numel()
386
+ total_numel += unpartitioned_numel
387
+
388
+ param_frags = tuple(model_state.frozen_param_fragments[name] for model_state in zero_model_states)
389
+ state_dict[name] = torch.cat(param_frags, 0).narrow(0, 0, unpartitioned_numel).view(shape)
390
+
391
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
392
+
393
+ if debug:
394
+ print(
395
+ f"Frozen params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
396
+ )
397
+
398
+ print(f"Reconstructed Frozen fp32 state dict with {total_params} params {total_numel} elements")
399
+
400
+
401
+ def _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states):
402
+ param_shapes = zero_model_states[0].param_shapes
403
+ avail_numel = fp32_flat_groups[0].numel() * world_size
404
+ # Reconstruction protocol: For zero3 we need to zip the partitions together at boundary of each
405
+ # param, re-consolidating each param, while dealing with padding if any
406
+
407
+ # merge list of dicts, preserving order
408
+ param_shapes = {k: v for d in param_shapes for k, v in d.items()}
409
+
410
+ if debug:
411
+ for i in range(world_size):
412
+ print(f"{FP32_FLAT_GROUPS}[{i}].shape={fp32_flat_groups[i].shape}")
413
+
414
+ wanted_params = len(param_shapes)
415
+ wanted_numel = sum(shape.numel() for shape in param_shapes.values())
416
+ # not asserting if there is a mismatch due to possible padding
417
+ avail_numel = fp32_flat_groups[0].numel() * world_size
418
+ print(f"Trainable params: Have {avail_numel} numels to process.")
419
+ print(f"Trainable params: Need {wanted_numel} numels in {wanted_params} params.")
420
+
421
+ # params
422
+ # XXX: for huge models that can't fit into the host's RAM we will have to recode this to support
423
+ # out-of-core computing solution
424
+ offset = 0
425
+ total_numel = 0
426
+ total_params = 0
427
+ for name, shape in tqdm(param_shapes.items(), desc='Gathering Sharded Weights'):
428
+ unpartitioned_numel = shape.numel()
429
+ total_numel += unpartitioned_numel
430
+ total_params += 1
431
+ partitioned_numel, partitioned_padding_numel = zero3_partitioned_param_info(unpartitioned_numel, world_size)
432
+
433
+ if debug:
434
+ print(
435
+ f"Trainable params: {total_params} {name} full shape: {shape} partition0 numel={partitioned_numel} partitioned_padding_numel={partitioned_padding_numel}"
436
+ )
437
+
438
+ # XXX: memory usage doubles here
439
+ state_dict[name] = torch.cat(
440
+ tuple(fp32_flat_groups[i].narrow(0, offset, partitioned_numel) for i in range(world_size)),
441
+ 0).narrow(0, 0, unpartitioned_numel).view(shape)
442
+ offset += partitioned_numel
443
+
444
+ offset *= world_size
445
+
446
+ # Sanity check
447
+ if offset != avail_numel:
448
+ raise ValueError(f"consumed {offset} numels out of {avail_numel} - something is wrong")
449
+
450
+ print(f"Reconstructed Trainable fp32 state dict with {total_params} params {total_numel} elements")
451
+
452
+
453
+ def _get_fp32_state_dict_from_zero3_checkpoint(world_size, fp32_flat_groups, zero_model_states,
454
+ exclude_frozen_parameters):
455
+ state_dict = OrderedDict()
456
+
457
+ # buffers
458
+ buffers = zero_model_states[0].buffers
459
+ state_dict.update(buffers)
460
+ if debug:
461
+ print(f"added {len(buffers)} buffers")
462
+
463
+ if not exclude_frozen_parameters:
464
+ _zero3_merge_frozen_params(state_dict, world_size, zero_model_states)
465
+
466
+ _zero3_merge_trainable_params(state_dict, world_size, fp32_flat_groups, zero_model_states)
467
+
468
+ # recover shared parameters
469
+ for pair in zero_model_states[0].shared_params:
470
+ if pair[1] in state_dict:
471
+ state_dict[pair[0]] = state_dict[pair[1]]
472
+
473
+ return state_dict
474
+
475
+
476
+ def get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag=None, exclude_frozen_parameters=False):
477
+ """
478
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated state_dict that can be loaded with
479
+ ``load_state_dict()`` and used for training without DeepSpeed or shared with others, for example
480
+ via a model hub.
481
+
482
+ Args:
483
+ - ``checkpoint_dir``: path to the desired checkpoint folder
484
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in 'latest' file. e.g., ``global_step14``
485
+ - ``exclude_frozen_parameters``: exclude frozen parameters
486
+
487
+ Returns:
488
+ - pytorch ``state_dict``
489
+
490
+ Note: this approach may not work if your application doesn't have sufficient free CPU memory and
491
+ you may need to use the offline approach using the ``zero_to_fp32.py`` script that is saved with
492
+ the checkpoint.
493
+
494
+ A typical usage might be ::
495
+
496
+ from deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint
497
+ # do the training and checkpoint saving
498
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir) # already on cpu
499
+ model = model.cpu() # move to cpu
500
+ model.load_state_dict(state_dict)
501
+ # submit to model hub or save the model to share with others
502
+
503
+ In this example the ``model`` will no longer be usable in the deepspeed context of the same
504
+ application. i.e. you will need to re-initialize the deepspeed engine, since
505
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
506
+
507
+ If you want it all done for you, use ``load_state_dict_from_zero_checkpoint`` instead.
508
+
509
+ """
510
+ if tag is None:
511
+ latest_path = os.path.join(checkpoint_dir, 'latest')
512
+ if os.path.isfile(latest_path):
513
+ with open(latest_path, 'r') as fd:
514
+ tag = fd.read().strip()
515
+ else:
516
+ raise ValueError(f"Unable to find 'latest' file at {latest_path}")
517
+
518
+ ds_checkpoint_dir = os.path.join(checkpoint_dir, tag)
519
+
520
+ if not os.path.isdir(ds_checkpoint_dir):
521
+ raise FileNotFoundError(f"Directory '{ds_checkpoint_dir}' doesn't exist")
522
+
523
+ return _get_fp32_state_dict_from_zero_checkpoint(ds_checkpoint_dir, exclude_frozen_parameters)
524
+
525
+
526
+ def convert_zero_checkpoint_to_fp32_state_dict(checkpoint_dir,
527
+ output_dir,
528
+ max_shard_size="5GB",
529
+ safe_serialization=False,
530
+ tag=None,
531
+ exclude_frozen_parameters=False):
532
+ """
533
+ Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict`` file that can be
534
+ loaded with ``torch.load(file)`` + ``load_state_dict()`` and used for training without DeepSpeed.
535
+
536
+ Args:
537
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
538
+ - ``output_dir``: directory to the pytorch fp32 state_dict output files
539
+ - ``max_shard_size``: the maximum size for a checkpoint before being sharded, default value is 5GB
540
+ - ``safe_serialization``: whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
541
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
542
+ - ``exclude_frozen_parameters``: exclude frozen parameters
543
+ """
544
+ # Dependency pre-check
545
+ if safe_serialization:
546
+ try:
547
+ from safetensors.torch import save_file
548
+ except ImportError:
549
+ print('If you want to use `safe_serialization`, please `pip install safetensors`')
550
+ raise
551
+ if max_shard_size is not None:
552
+ try:
553
+ from huggingface_hub import split_torch_state_dict_into_shards
554
+ except ImportError:
555
+ print('If you want to use `max_shard_size`, please `pip install huggingface_hub`')
556
+ raise
557
+
558
+ # Convert zero checkpoint to state_dict
559
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag, exclude_frozen_parameters)
560
+
561
+ # Shard the model if it is too big.
562
+ weights_name = "model.safetensors" if safe_serialization else "pytorch_model.bin"
563
+ if max_shard_size is not None:
564
+ filename_pattern = weights_name.replace(".bin", "{suffix}.bin").replace(".safetensors", "{suffix}.safetensors")
565
+ state_dict_split = split_torch_state_dict_into_shards(state_dict,
566
+ filename_pattern=filename_pattern,
567
+ max_shard_size=max_shard_size)
568
+ else:
569
+ from collections import namedtuple
570
+ StateDictSplit = namedtuple("StateDictSplit", ["is_sharded", "filename_to_tensors"])
571
+ state_dict_split = StateDictSplit(is_sharded=False,
572
+ filename_to_tensors={weights_name: list(state_dict.keys())})
573
+
574
+ # Save the model
575
+ filename_to_tensors = state_dict_split.filename_to_tensors.items()
576
+ for shard_file, tensors in tqdm(filename_to_tensors, desc="Saving checkpoint shards"):
577
+ shard = {tensor: state_dict[tensor].contiguous() for tensor in tensors}
578
+ output_path = os.path.join(output_dir, shard_file)
579
+ if safe_serialization:
580
+ save_file(shard, output_path, metadata={"format": "pt"})
581
+ else:
582
+ torch.save(shard, output_path)
583
+
584
+ # Save index if sharded
585
+ if state_dict_split.is_sharded:
586
+ index = {
587
+ "metadata": state_dict_split.metadata,
588
+ "weight_map": state_dict_split.tensor_to_filename,
589
+ }
590
+ save_index_file = "model.safetensors.index.json" if safe_serialization else "pytorch_model.bin.index.json"
591
+ save_index_file = os.path.join(output_dir, save_index_file)
592
+ with open(save_index_file, "w", encoding="utf-8") as f:
593
+ content = json.dumps(index, indent=2, sort_keys=True) + "\n"
594
+ f.write(content)
595
+
596
+
597
+ def load_state_dict_from_zero_checkpoint(model, checkpoint_dir, tag=None):
598
+ """
599
+ 1. Put the provided model to cpu
600
+ 2. Convert ZeRO 2 or 3 checkpoint into a single fp32 consolidated ``state_dict``
601
+ 3. Load it into the provided model
602
+
603
+ Args:
604
+ - ``model``: the model object to update
605
+ - ``checkpoint_dir``: path to the desired checkpoint folder. (one that contains the tag-folder, like ``global_step14``)
606
+ - ``tag``: checkpoint tag used as a unique identifier for checkpoint. If not provided will attempt to load tag in the file named ``latest`` in the checkpoint folder, e.g., ``global_step14``
607
+
608
+ Returns:
609
+ - ``model`: modified model
610
+
611
+ Make sure you have plenty of CPU memory available before you call this function. If you don't
612
+ have enough use the ``zero_to_fp32.py`` utility to do the conversion. You will find it
613
+ conveniently placed for you in the checkpoint folder.
614
+
615
+ A typical usage might be ::
616
+
617
+ from deepspeed.utils.zero_to_fp32 import load_state_dict_from_zero_checkpoint
618
+ model = load_state_dict_from_zero_checkpoint(trainer.model, checkpoint_dir)
619
+ # submit to model hub or save the model to share with others
620
+
621
+ Note, that once this was run, the ``model`` will no longer be usable in the deepspeed context
622
+ of the same application. i.e. you will need to re-initialize the deepspeed engine, since
623
+ ``model.load_state_dict(state_dict)`` will remove all the deepspeed magic from it.
624
+
625
+ """
626
+ logger.info(f"Extracting fp32 weights")
627
+ state_dict = get_fp32_state_dict_from_zero_checkpoint(checkpoint_dir, tag)
628
+
629
+ logger.info(f"Overwriting model with fp32 weights")
630
+ model = model.cpu()
631
+ model.load_state_dict(state_dict, strict=False)
632
+
633
+ return model
634
+
635
+
636
+ if __name__ == "__main__":
637
+ parser = argparse.ArgumentParser()
638
+ parser.add_argument("checkpoint_dir",
639
+ type=str,
640
+ help="path to the desired checkpoint folder, e.g., path/checkpoint-12")
641
+ parser.add_argument("output_dir",
642
+ type=str,
643
+ help="directory to the pytorch fp32 state_dict output files"
644
+ "(e.g. path/checkpoint-12-output/)")
645
+ parser.add_argument(
646
+ "--max_shard_size",
647
+ type=str,
648
+ default="5GB",
649
+ help="The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size"
650
+ "lower than this size. If expressed as a string, needs to be digits followed by a unit (like `5MB`"
651
+ "We default it to 5GB in order for models to be able to run easily on free-tier google colab instances"
652
+ "without CPU OOM issues.")
653
+ parser.add_argument(
654
+ "--safe_serialization",
655
+ default=False,
656
+ action='store_true',
657
+ help="Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).")
658
+ parser.add_argument("-t",
659
+ "--tag",
660
+ type=str,
661
+ default=None,
662
+ help="checkpoint tag used as a unique identifier for checkpoint. e.g., global_step1")
663
+ parser.add_argument("--exclude_frozen_parameters", action='store_true', help="exclude frozen parameters")
664
+ parser.add_argument("-d", "--debug", action='store_true', help="enable debug")
665
+ args = parser.parse_args()
666
+
667
+ debug = args.debug
668
+
669
+ convert_zero_checkpoint_to_fp32_state_dict(args.checkpoint_dir,
670
+ args.output_dir,
671
+ max_shard_size=args.max_shard_size,
672
+ safe_serialization=args.safe_serialization,
673
+ tag=args.tag,
674
+ exclude_frozen_parameters=args.exclude_frozen_parameters)