Hugging Face's logo

Taywon
/
B2minus_v4_e3

Text Generation
PEFT
Safetensors
Transformers
lora
sft
trl
conversational
Model card Files
xet
 Community
Taywon commited on
Commit
080c478
·
verified ·
1 Parent(s): fc4bf25

Upload folder using huggingface_hub

Browse files
Files changed (8) hide show
  1. .gitattributes +1 -0
  2. README.md +209 -0
  3. adapter_config.json +48 -0
  4. adapter_model.safetensors +3 -0
  5. chat_template.jinja +109 -0
  6. tokenizer.json +3 -0
  7. tokenizer_config.json +15 -0
  8. trainer_state.json +3154 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: meta-llama/Llama-3.1-70B-Instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:meta-llama/Llama-3.1-70B-Instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ ---
12
+
13
+ # Model Card for Model ID
14
+
15
+ <!-- Provide a quick summary of what the model is/does. -->
16
+
17
+
18
+
19
+ ## Model Details
20
+
21
+ ### Model Description
22
+
23
+ <!-- Provide a longer summary of what this model is. -->
24
+
25
+
26
+
27
+ - **Developed by:** [More Information Needed]
28
+ - **Funded by [optional]:** [More Information Needed]
29
+ - **Shared by [optional]:** [More Information Needed]
30
+ - **Model type:** [More Information Needed]
31
+ - **Language(s) (NLP):** [More Information Needed]
32
+ - **License:** [More Information Needed]
33
+ - **Finetuned from model [optional]:** [More Information Needed]
34
+
35
+ ### Model Sources [optional]
36
+
37
+ <!-- Provide the basic links for the model. -->
38
+
39
+ - **Repository:** [More Information Needed]
40
+ - **Paper [optional]:** [More Information Needed]
41
+ - **Demo [optional]:** [More Information Needed]
42
+
43
+ ## Uses
44
+
45
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
46
+
47
+ ### Direct Use
48
+
49
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
50
+
51
+ [More Information Needed]
52
+
53
+ ### Downstream Use [optional]
54
+
55
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
56
+
57
+ [More Information Needed]
58
+
59
+ ### Out-of-Scope Use
60
+
61
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
62
+
63
+ [More Information Needed]
64
+
65
+ ## Bias, Risks, and Limitations
66
+
67
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
68
+
69
+ [More Information Needed]
70
+
71
+ ### Recommendations
72
+
73
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
74
+
75
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
76
+
77
+ ## How to Get Started with the Model
78
+
79
+ Use the code below to get started with the model.
80
+
81
+ [More Information Needed]
82
+
83
+ ## Training Details
84
+
85
+ ### Training Data
86
+
87
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
88
+
89
+ [More Information Needed]
90
+
91
+ ### Training Procedure
92
+
93
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
94
+
95
+ #### Preprocessing [optional]
96
+
97
+ [More Information Needed]
98
+
99
+
100
+ #### Training Hyperparameters
101
+
102
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
103
+
104
+ #### Speeds, Sizes, Times [optional]
105
+
106
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
107
+
108
+ [More Information Needed]
109
+
110
+ ## Evaluation
111
+
112
+ <!-- This section describes the evaluation protocols and provides the results. -->
113
+
114
+ ### Testing Data, Factors & Metrics
115
+
116
+ #### Testing Data
117
+
118
+ <!-- This should link to a Dataset Card if possible. -->
119
+
120
+ [More Information Needed]
121
+
122
+ #### Factors
123
+
124
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
125
+
126
+ [More Information Needed]
127
+
128
+ #### Metrics
129
+
130
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
131
+
132
+ [More Information Needed]
133
+
134
+ ### Results
135
+
136
+ [More Information Needed]
137
+
138
+ #### Summary
139
+
140
+
141
+
142
+ ## Model Examination [optional]
143
+
144
+ <!-- Relevant interpretability work for the model goes here -->
145
+
146
+ [More Information Needed]
147
+
148
+ ## Environmental Impact
149
+
150
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
151
+
152
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
153
+
154
+ - **Hardware Type:** [More Information Needed]
155
+ - **Hours used:** [More Information Needed]
156
+ - **Cloud Provider:** [More Information Needed]
157
+ - **Compute Region:** [More Information Needed]
158
+ - **Carbon Emitted:** [More Information Needed]
159
+
160
+ ## Technical Specifications [optional]
161
+
162
+ ### Model Architecture and Objective
163
+
164
+ [More Information Needed]
165
+
166
+ ### Compute Infrastructure
167
+
168
+ [More Information Needed]
169
+
170
+ #### Hardware
171
+
172
+ [More Information Needed]
173
+
174
+ #### Software
175
+
176
+ [More Information Needed]
177
+
178
+ ## Citation [optional]
179
+
180
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
181
+
182
+ **BibTeX:**
183
+
184
+ [More Information Needed]
185
+
186
+ **APA:**
187
+
188
+ [More Information Needed]
189
+
190
+ ## Glossary [optional]
191
+
192
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
193
+
194
+ [More Information Needed]
195
+
196
+ ## More Information [optional]
197
+
198
+ [More Information Needed]
199
+
200
+ ## Model Card Authors [optional]
201
+
202
+ [More Information Needed]
203
+
204
+ ## Model Card Contact
205
+
206
+ [More Information Needed]
207
+ ### Framework versions
208
+
209
+ - PEFT 0.19.1
adapter_config.json ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "meta-llama/Llama-3.1-70B-Instruct",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": false,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 128,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "lora_ga_config": null,
23
+ "megatron_config": null,
24
+ "megatron_core": "megatron.core",
25
+ "modules_to_save": null,
26
+ "peft_type": "LORA",
27
+ "peft_version": "0.18.1",
28
+ "qalora_group_size": 16,
29
+ "r": 64,
30
+ "rank_pattern": {},
31
+ "revision": null,
32
+ "target_modules": [
33
+ "gate_proj",
34
+ "v_proj",
35
+ "down_proj",
36
+ "q_proj",
37
+ "k_proj",
38
+ "up_proj",
39
+ "o_proj"
40
+ ],
41
+ "target_parameters": null,
42
+ "task_type": "CAUSAL_LM",
43
+ "trainable_token_indices": null,
44
+ "use_bdlora": null,
45
+ "use_dora": false,
46
+ "use_qalora": false,
47
+ "use_rslora": false
48
+ }
adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9be679b16b02671c859c0eb65d3fba40358d6b27f9aff782c7eca40f1a825c1
3
+ size 3313653480
chat_template.jinja ADDED
@@ -0,0 +1,109 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {{- bos_token }}
2
+ {%- if custom_tools is defined %}
3
+ {%- set tools = custom_tools %}
4
+ {%- endif %}
5
+ {%- if not tools_in_user_message is defined %}
6
+ {%- set tools_in_user_message = true %}
7
+ {%- endif %}
8
+ {%- if not date_string is defined %}
9
+ {%- set date_string = "26 Jul 2024" %}
10
+ {%- endif %}
11
+ {%- if not tools is defined %}
12
+ {%- set tools = none %}
13
+ {%- endif %}
14
+
15
+ {#- This block extracts the system message, so we can slot it into the right place. #}
16
+ {%- if messages[0]['role'] == 'system' %}
17
+ {%- set system_message = messages[0]['content']|trim %}
18
+ {%- set messages = messages[1:] %}
19
+ {%- else %}
20
+ {%- set system_message = "" %}
21
+ {%- endif %}
22
+
23
+ {#- System message + builtin tools #}
24
+ {{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
25
+ {%- if builtin_tools is defined or tools is not none %}
26
+ {{- "Environment: ipython\n" }}
27
+ {%- endif %}
28
+ {%- if builtin_tools is defined %}
29
+ {{- "Tools: " + builtin_tools | reject('equalto', 'code_interpreter') | join(", ") + "\n\n"}}
30
+ {%- endif %}
31
+ {{- "Cutting Knowledge Date: December 2023\n" }}
32
+ {{- "Today Date: " + date_string + "\n\n" }}
33
+ {%- if tools is not none and not tools_in_user_message %}
34
+ {{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
35
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
36
+ {{- "Do not use variables.\n\n" }}
37
+ {%- for t in tools %}
38
+ {{- t | tojson(indent=4) }}
39
+ {{- "\n\n" }}
40
+ {%- endfor %}
41
+ {%- endif %}
42
+ {{- system_message }}
43
+ {{- "<|eot_id|>" }}
44
+
45
+ {#- Custom tools are passed in a user message with some extra guidance #}
46
+ {%- if tools_in_user_message and not tools is none %}
47
+ {#- Extract the first user message so we can plug it in here #}
48
+ {%- if messages | length != 0 %}
49
+ {%- set first_user_message = messages[0]['content']|trim %}
50
+ {%- set messages = messages[1:] %}
51
+ {%- else %}
52
+ {{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
53
+ {%- endif %}
54
+ {{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
55
+ {{- "Given the following functions, please respond with a JSON for a function call " }}
56
+ {{- "with its proper arguments that best answers the given prompt.\n\n" }}
57
+ {{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
58
+ {{- "Do not use variables.\n\n" }}
59
+ {%- for t in tools %}
60
+ {{- t | tojson(indent=4) }}
61
+ {{- "\n\n" }}
62
+ {%- endfor %}
63
+ {{- first_user_message + "<|eot_id|>"}}
64
+ {%- endif %}
65
+
66
+ {%- for message in messages %}
67
+ {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
68
+ {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
69
+ {%- elif 'tool_calls' in message %}
70
+ {%- if not message.tool_calls|length == 1 %}
71
+ {{- raise_exception("This model only supports single tool-calls at once!") }}
72
+ {%- endif %}
73
+ {%- set tool_call = message.tool_calls[0].function %}
74
+ {%- if builtin_tools is defined and tool_call.name in builtin_tools %}
75
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
76
+ {{- "<|python_tag|>" + tool_call.name + ".call(" }}
77
+ {%- for arg_name, arg_val in tool_call.arguments | items %}
78
+ {{- arg_name + '="' + arg_val + '"' }}
79
+ {%- if not loop.last %}
80
+ {{- ", " }}
81
+ {%- endif %}
82
+ {%- endfor %}
83
+ {{- ")" }}
84
+ {%- else %}
85
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
86
+ {{- '{"name": "' + tool_call.name + '", ' }}
87
+ {{- '"parameters": ' }}
88
+ {{- tool_call.arguments | tojson }}
89
+ {{- "}" }}
90
+ {%- endif %}
91
+ {%- if builtin_tools is defined %}
92
+ {#- This means we're in ipython mode #}
93
+ {{- "<|eom_id|>" }}
94
+ {%- else %}
95
+ {{- "<|eot_id|>" }}
96
+ {%- endif %}
97
+ {%- elif message.role == "tool" or message.role == "ipython" %}
98
+ {{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
99
+ {%- if message.content is mapping or message.content is iterable %}
100
+ {{- message.content | tojson }}
101
+ {%- else %}
102
+ {{- message.content }}
103
+ {%- endif %}
104
+ {{- "<|eot_id|>" }}
105
+ {%- endif %}
106
+ {%- endfor %}
107
+ {%- if add_generation_prompt %}
108
+ {{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
109
+ {%- endif %}
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
3
+ size 17209920
tokenizer_config.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<|begin_of_text|>",
4
+ "clean_up_tokenization_spaces": true,
5
+ "eos_token": "<|eot_id|>",
6
+ "is_local": false,
7
+ "local_files_only": false,
8
+ "model_input_names": [
9
+ "input_ids",
10
+ "attention_mask"
11
+ ],
12
+ "model_max_length": 131072,
13
+ "pad_token": "<|eot_id|>",
14
+ "tokenizer_class": "TokenizersBackend"
15
+ }
trainer_state.json ADDED
@@ -0,0 +1,3154 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 3.0,
6
+ "eval_steps": 500,
7
+ "global_step": 312,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "entropy": 1.2831476479768753,
14
+ "epoch": 0.009685230024213076,
15
+ "grad_norm": 1.257500410079956,
16
+ "learning_rate": 0.0,
17
+ "loss": 2.0216622352600098,
18
+ "mean_token_accuracy": 0.5382094085216522,
19
+ "num_tokens": 97485.0,
20
+ "step": 1
21
+ },
22
+ {
23
+ "entropy": 1.2757050544023514,
24
+ "epoch": 0.01937046004842615,
25
+ "grad_norm": 1.3158080577850342,
26
+ "learning_rate": 3.125e-06,
27
+ "loss": 2.071122646331787,
28
+ "mean_token_accuracy": 0.5314131379127502,
29
+ "num_tokens": 194184.0,
30
+ "step": 2
31
+ },
32
+ {
33
+ "entropy": 1.2694316655397415,
34
+ "epoch": 0.029055690072639227,
35
+ "grad_norm": 1.1289395093917847,
36
+ "learning_rate": 6.25e-06,
37
+ "loss": 1.9294486045837402,
38
+ "mean_token_accuracy": 0.5518109127879143,
39
+ "num_tokens": 287660.0,
40
+ "step": 3
41
+ },
42
+ {
43
+ "entropy": 1.3956594914197922,
44
+ "epoch": 0.0387409200968523,
45
+ "grad_norm": 0.9914306998252869,
46
+ "learning_rate": 9.375000000000001e-06,
47
+ "loss": 1.941465139389038,
48
+ "mean_token_accuracy": 0.54258743673563,
49
+ "num_tokens": 384672.0,
50
+ "step": 4
51
+ },
52
+ {
53
+ "entropy": 1.3979678750038147,
54
+ "epoch": 0.048426150121065374,
55
+ "grad_norm": 0.7685282230377197,
56
+ "learning_rate": 1.25e-05,
57
+ "loss": 1.8111070394515991,
58
+ "mean_token_accuracy": 0.5647609829902649,
59
+ "num_tokens": 481921.0,
60
+ "step": 5
61
+ },
62
+ {
63
+ "entropy": 1.6170997768640518,
64
+ "epoch": 0.05811138014527845,
65
+ "grad_norm": 0.5168716311454773,
66
+ "learning_rate": 1.5625e-05,
67
+ "loss": 1.7966232299804688,
68
+ "mean_token_accuracy": 0.5577141717076302,
69
+ "num_tokens": 577866.0,
70
+ "step": 6
71
+ },
72
+ {
73
+ "entropy": 1.6874641478061676,
74
+ "epoch": 0.06779661016949153,
75
+ "grad_norm": 0.4072948098182678,
76
+ "learning_rate": 1.8750000000000002e-05,
77
+ "loss": 1.708348035812378,
78
+ "mean_token_accuracy": 0.5744073167443275,
79
+ "num_tokens": 674698.0,
80
+ "step": 7
81
+ },
82
+ {
83
+ "entropy": 1.7709944546222687,
84
+ "epoch": 0.0774818401937046,
85
+ "grad_norm": 0.4375021159648895,
86
+ "learning_rate": 2.1875e-05,
87
+ "loss": 1.6493638753890991,
88
+ "mean_token_accuracy": 0.5816963836550713,
89
+ "num_tokens": 771175.0,
90
+ "step": 8
91
+ },
92
+ {
93
+ "entropy": 1.869639590382576,
94
+ "epoch": 0.08716707021791767,
95
+ "grad_norm": 0.5281513333320618,
96
+ "learning_rate": 2.5e-05,
97
+ "loss": 1.6473325490951538,
98
+ "mean_token_accuracy": 0.5845584571361542,
99
+ "num_tokens": 868470.0,
100
+ "step": 9
101
+ },
102
+ {
103
+ "entropy": 1.8225883692502975,
104
+ "epoch": 0.09685230024213075,
105
+ "grad_norm": 0.5096434354782104,
106
+ "learning_rate": 2.8125000000000003e-05,
107
+ "loss": 1.590818166732788,
108
+ "mean_token_accuracy": 0.5921354293823242,
109
+ "num_tokens": 964784.0,
110
+ "step": 10
111
+ },
112
+ {
113
+ "entropy": 1.7066199630498886,
114
+ "epoch": 0.10653753026634383,
115
+ "grad_norm": 0.4193650186061859,
116
+ "learning_rate": 3.125e-05,
117
+ "loss": 1.529557228088379,
118
+ "mean_token_accuracy": 0.5996061563491821,
119
+ "num_tokens": 1061836.0,
120
+ "step": 11
121
+ },
122
+ {
123
+ "entropy": 1.6335420161485672,
124
+ "epoch": 0.1162227602905569,
125
+ "grad_norm": 0.34470972418785095,
126
+ "learning_rate": 3.4375e-05,
127
+ "loss": 1.5362547636032104,
128
+ "mean_token_accuracy": 0.5929826945066452,
129
+ "num_tokens": 1158471.0,
130
+ "step": 12
131
+ },
132
+ {
133
+ "entropy": 1.4438551515340805,
134
+ "epoch": 0.12590799031476999,
135
+ "grad_norm": 0.27761033177375793,
136
+ "learning_rate": 3.7500000000000003e-05,
137
+ "loss": 1.4457957744598389,
138
+ "mean_token_accuracy": 0.6143342331051826,
139
+ "num_tokens": 1255084.0,
140
+ "step": 13
141
+ },
142
+ {
143
+ "entropy": 1.392322763800621,
144
+ "epoch": 0.13559322033898305,
145
+ "grad_norm": 0.2844429016113281,
146
+ "learning_rate": 4.0625000000000005e-05,
147
+ "loss": 1.4511373043060303,
148
+ "mean_token_accuracy": 0.6104418113827705,
149
+ "num_tokens": 1352697.0,
150
+ "step": 14
151
+ },
152
+ {
153
+ "entropy": 1.3371251970529556,
154
+ "epoch": 0.14527845036319612,
155
+ "grad_norm": 0.2822510898113251,
156
+ "learning_rate": 4.375e-05,
157
+ "loss": 1.4322829246520996,
158
+ "mean_token_accuracy": 0.6155784726142883,
159
+ "num_tokens": 1450346.0,
160
+ "step": 15
161
+ },
162
+ {
163
+ "entropy": 1.2996839582920074,
164
+ "epoch": 0.1549636803874092,
165
+ "grad_norm": 0.2731319069862366,
166
+ "learning_rate": 4.6875e-05,
167
+ "loss": 1.399651288986206,
168
+ "mean_token_accuracy": 0.6181296408176422,
169
+ "num_tokens": 1546056.0,
170
+ "step": 16
171
+ },
172
+ {
173
+ "entropy": 1.286782756447792,
174
+ "epoch": 0.16464891041162227,
175
+ "grad_norm": 0.2287229746580124,
176
+ "learning_rate": 5e-05,
177
+ "loss": 1.3414286375045776,
178
+ "mean_token_accuracy": 0.6305811852216721,
179
+ "num_tokens": 1644072.0,
180
+ "step": 17
181
+ },
182
+ {
183
+ "entropy": 1.2884749919176102,
184
+ "epoch": 0.17433414043583534,
185
+ "grad_norm": 0.24129998683929443,
186
+ "learning_rate": 4.999859193643945e-05,
187
+ "loss": 1.2895803451538086,
188
+ "mean_token_accuracy": 0.6411498486995697,
189
+ "num_tokens": 1740823.0,
190
+ "step": 18
191
+ },
192
+ {
193
+ "entropy": 1.399221807718277,
194
+ "epoch": 0.18401937046004843,
195
+ "grad_norm": 0.24152599275112152,
196
+ "learning_rate": 4.999436790436924e-05,
197
+ "loss": 1.3620624542236328,
198
+ "mean_token_accuracy": 0.6247868910431862,
199
+ "num_tokens": 1837872.0,
200
+ "step": 19
201
+ },
202
+ {
203
+ "entropy": 1.3781700730323792,
204
+ "epoch": 0.1937046004842615,
205
+ "grad_norm": 0.22794979810714722,
206
+ "learning_rate": 4.9987328379605816e-05,
207
+ "loss": 1.3193318843841553,
208
+ "mean_token_accuracy": 0.6357730701565742,
209
+ "num_tokens": 1935638.0,
210
+ "step": 20
211
+ },
212
+ {
213
+ "entropy": 1.3448941111564636,
214
+ "epoch": 0.2033898305084746,
215
+ "grad_norm": 0.20295539498329163,
216
+ "learning_rate": 4.9977474155117045e-05,
217
+ "loss": 1.2904666662216187,
218
+ "mean_token_accuracy": 0.6384864747524261,
219
+ "num_tokens": 2033151.0,
220
+ "step": 21
221
+ },
222
+ {
223
+ "entropy": 1.3269115835428238,
224
+ "epoch": 0.21307506053268765,
225
+ "grad_norm": 0.19907879829406738,
226
+ "learning_rate": 4.9964806340932865e-05,
227
+ "loss": 1.2992527484893799,
228
+ "mean_token_accuracy": 0.6387153342366219,
229
+ "num_tokens": 2130350.0,
230
+ "step": 22
231
+ },
232
+ {
233
+ "entropy": 1.2736097276210785,
234
+ "epoch": 0.22276029055690072,
235
+ "grad_norm": 0.1993221640586853,
236
+ "learning_rate": 4.9949326364020314e-05,
237
+ "loss": 1.2424592971801758,
238
+ "mean_token_accuracy": 0.6515100449323654,
239
+ "num_tokens": 2226505.0,
240
+ "step": 23
241
+ },
242
+ {
243
+ "entropy": 1.2606256753206253,
244
+ "epoch": 0.2324455205811138,
245
+ "grad_norm": 0.1928444802761078,
246
+ "learning_rate": 4.993103596812268e-05,
247
+ "loss": 1.2370142936706543,
248
+ "mean_token_accuracy": 0.6512897834181786,
249
+ "num_tokens": 2324337.0,
250
+ "step": 24
251
+ },
252
+ {
253
+ "entropy": 1.1828058958053589,
254
+ "epoch": 0.24213075060532688,
255
+ "grad_norm": 0.1695406287908554,
256
+ "learning_rate": 4.9909937213563165e-05,
257
+ "loss": 1.1700199842453003,
258
+ "mean_token_accuracy": 0.6674213483929634,
259
+ "num_tokens": 2419418.0,
260
+ "step": 25
261
+ },
262
+ {
263
+ "entropy": 1.2690571248531342,
264
+ "epoch": 0.25181598062953997,
265
+ "grad_norm": 0.17888334393501282,
266
+ "learning_rate": 4.988603247701277e-05,
267
+ "loss": 1.2402149438858032,
268
+ "mean_token_accuracy": 0.6481731608510017,
269
+ "num_tokens": 2515666.0,
270
+ "step": 26
271
+ },
272
+ {
273
+ "entropy": 1.2278835028409958,
274
+ "epoch": 0.26150121065375304,
275
+ "grad_norm": 0.1878158003091812,
276
+ "learning_rate": 4.985932445122257e-05,
277
+ "loss": 1.2124648094177246,
278
+ "mean_token_accuracy": 0.6553028896450996,
279
+ "num_tokens": 2613197.0,
280
+ "step": 27
281
+ },
282
+ {
283
+ "entropy": 1.2094718217849731,
284
+ "epoch": 0.2711864406779661,
285
+ "grad_norm": 0.1670471876859665,
286
+ "learning_rate": 4.982981614472039e-05,
287
+ "loss": 1.2036645412445068,
288
+ "mean_token_accuracy": 0.6590213552117348,
289
+ "num_tokens": 2709760.0,
290
+ "step": 28
291
+ },
292
+ {
293
+ "entropy": 1.1803771257400513,
294
+ "epoch": 0.28087167070217917,
295
+ "grad_norm": 0.16492998600006104,
296
+ "learning_rate": 4.979751088147192e-05,
297
+ "loss": 1.1872878074645996,
298
+ "mean_token_accuracy": 0.661787860095501,
299
+ "num_tokens": 2807308.0,
300
+ "step": 29
301
+ },
302
+ {
303
+ "entropy": 1.170692041516304,
304
+ "epoch": 0.29055690072639223,
305
+ "grad_norm": 0.17398205399513245,
306
+ "learning_rate": 4.97624123005063e-05,
307
+ "loss": 1.1832412481307983,
308
+ "mean_token_accuracy": 0.661918580532074,
309
+ "num_tokens": 2904379.0,
310
+ "step": 30
311
+ },
312
+ {
313
+ "entropy": 1.1982254832983017,
314
+ "epoch": 0.30024213075060535,
315
+ "grad_norm": 0.16584837436676025,
316
+ "learning_rate": 4.972452435550614e-05,
317
+ "loss": 1.2053539752960205,
318
+ "mean_token_accuracy": 0.6568507477641106,
319
+ "num_tokens": 3001932.0,
320
+ "step": 31
321
+ },
322
+ {
323
+ "entropy": 1.139011174440384,
324
+ "epoch": 0.3099273607748184,
325
+ "grad_norm": 0.16256465017795563,
326
+ "learning_rate": 4.968385131436222e-05,
327
+ "loss": 1.1371057033538818,
328
+ "mean_token_accuracy": 0.6699384972453117,
329
+ "num_tokens": 3099859.0,
330
+ "step": 32
331
+ },
332
+ {
333
+ "entropy": 1.1760681718587875,
334
+ "epoch": 0.3196125907990315,
335
+ "grad_norm": 0.15591397881507874,
336
+ "learning_rate": 4.9640397758692715e-05,
337
+ "loss": 1.160001277923584,
338
+ "mean_token_accuracy": 0.6634443625807762,
339
+ "num_tokens": 3197294.0,
340
+ "step": 33
341
+ },
342
+ {
343
+ "entropy": 1.1871805638074875,
344
+ "epoch": 0.32929782082324455,
345
+ "grad_norm": 0.1550437957048416,
346
+ "learning_rate": 4.9594168583327094e-05,
347
+ "loss": 1.1692184209823608,
348
+ "mean_token_accuracy": 0.6639450341463089,
349
+ "num_tokens": 3294768.0,
350
+ "step": 34
351
+ },
352
+ {
353
+ "entropy": 1.1860681176185608,
354
+ "epoch": 0.3389830508474576,
355
+ "grad_norm": 0.15519270300865173,
356
+ "learning_rate": 4.954516899575473e-05,
357
+ "loss": 1.1605687141418457,
358
+ "mean_token_accuracy": 0.6660626903176308,
359
+ "num_tokens": 3389702.0,
360
+ "step": 35
361
+ },
362
+ {
363
+ "entropy": 1.144161432981491,
364
+ "epoch": 0.3486682808716707,
365
+ "grad_norm": 0.16386644542217255,
366
+ "learning_rate": 4.949340451553833e-05,
367
+ "loss": 1.1134624481201172,
368
+ "mean_token_accuracy": 0.6774730086326599,
369
+ "num_tokens": 3486985.0,
370
+ "step": 36
371
+ },
372
+ {
373
+ "entropy": 1.169129118323326,
374
+ "epoch": 0.3583535108958838,
375
+ "grad_norm": 0.15737073123455048,
376
+ "learning_rate": 4.943888097369216e-05,
377
+ "loss": 1.1533232927322388,
378
+ "mean_token_accuracy": 0.6671040058135986,
379
+ "num_tokens": 3584177.0,
380
+ "step": 37
381
+ },
382
+ {
383
+ "entropy": 1.1375333964824677,
384
+ "epoch": 0.36803874092009686,
385
+ "grad_norm": 0.16654150187969208,
386
+ "learning_rate": 4.938160451202521e-05,
387
+ "loss": 1.1330571174621582,
388
+ "mean_token_accuracy": 0.6710135564208031,
389
+ "num_tokens": 3681733.0,
390
+ "step": 38
391
+ },
392
+ {
393
+ "entropy": 1.115285873413086,
394
+ "epoch": 0.37772397094430993,
395
+ "grad_norm": 0.15554173290729523,
396
+ "learning_rate": 4.9321581582449365e-05,
397
+ "loss": 1.1185455322265625,
398
+ "mean_token_accuracy": 0.6746590584516525,
399
+ "num_tokens": 3777606.0,
400
+ "step": 39
401
+ },
402
+ {
403
+ "entropy": 1.1352528482675552,
404
+ "epoch": 0.387409200968523,
405
+ "grad_norm": 0.15042954683303833,
406
+ "learning_rate": 4.9258818946252624e-05,
407
+ "loss": 1.125117540359497,
408
+ "mean_token_accuracy": 0.6720233038067818,
409
+ "num_tokens": 3874534.0,
410
+ "step": 40
411
+ },
412
+ {
413
+ "entropy": 1.1261803954839706,
414
+ "epoch": 0.39709443099273606,
415
+ "grad_norm": 0.1618603765964508,
416
+ "learning_rate": 4.9193323673337476e-05,
417
+ "loss": 1.1155072450637817,
418
+ "mean_token_accuracy": 0.6748175323009491,
419
+ "num_tokens": 3972562.0,
420
+ "step": 41
421
+ },
422
+ {
423
+ "entropy": 1.1054729223251343,
424
+ "epoch": 0.4067796610169492,
425
+ "grad_norm": 0.1928945779800415,
426
+ "learning_rate": 4.912510314142448e-05,
427
+ "loss": 1.0943855047225952,
428
+ "mean_token_accuracy": 0.6810023561120033,
429
+ "num_tokens": 4069253.0,
430
+ "step": 42
431
+ },
432
+ {
433
+ "entropy": 1.1252695173025131,
434
+ "epoch": 0.41646489104116224,
435
+ "grad_norm": 0.16767306625843048,
436
+ "learning_rate": 4.9054165035221236e-05,
437
+ "loss": 1.1221925020217896,
438
+ "mean_token_accuracy": 0.6732571944594383,
439
+ "num_tokens": 4167434.0,
440
+ "step": 43
441
+ },
442
+ {
443
+ "entropy": 1.0451640114188194,
444
+ "epoch": 0.4261501210653753,
445
+ "grad_norm": 0.15018437802791595,
446
+ "learning_rate": 4.898051734555676e-05,
447
+ "loss": 1.0456998348236084,
448
+ "mean_token_accuracy": 0.6917429268360138,
449
+ "num_tokens": 4263989.0,
450
+ "step": 44
451
+ },
452
+ {
453
+ "entropy": 1.0243945717811584,
454
+ "epoch": 0.4358353510895884,
455
+ "grad_norm": 0.15813371539115906,
456
+ "learning_rate": 4.890416836848127e-05,
457
+ "loss": 1.019399642944336,
458
+ "mean_token_accuracy": 0.6970273479819298,
459
+ "num_tokens": 4358172.0,
460
+ "step": 45
461
+ },
462
+ {
463
+ "entropy": 1.0881676077842712,
464
+ "epoch": 0.44552058111380144,
465
+ "grad_norm": 0.1647815704345703,
466
+ "learning_rate": 4.88251267043318e-05,
467
+ "loss": 1.089061975479126,
468
+ "mean_token_accuracy": 0.6811036169528961,
469
+ "num_tokens": 4454874.0,
470
+ "step": 46
471
+ },
472
+ {
473
+ "entropy": 1.1170424669981003,
474
+ "epoch": 0.4552058111380145,
475
+ "grad_norm": 0.15906141698360443,
476
+ "learning_rate": 4.87434012567633e-05,
477
+ "loss": 1.1157368421554565,
478
+ "mean_token_accuracy": 0.6773378178477287,
479
+ "num_tokens": 4551716.0,
480
+ "step": 47
481
+ },
482
+ {
483
+ "entropy": 1.1030111014842987,
484
+ "epoch": 0.4648910411622276,
485
+ "grad_norm": 0.1497914344072342,
486
+ "learning_rate": 4.865900123174574e-05,
487
+ "loss": 1.0889781713485718,
488
+ "mean_token_accuracy": 0.6786394864320755,
489
+ "num_tokens": 4649881.0,
490
+ "step": 48
491
+ },
492
+ {
493
+ "entropy": 1.1213176995515823,
494
+ "epoch": 0.4745762711864407,
495
+ "grad_norm": 0.14933931827545166,
496
+ "learning_rate": 4.857193613652711e-05,
497
+ "loss": 1.102480173110962,
498
+ "mean_token_accuracy": 0.6778694689273834,
499
+ "num_tokens": 4746560.0,
500
+ "step": 49
501
+ },
502
+ {
503
+ "entropy": 1.1182024255394936,
504
+ "epoch": 0.48426150121065376,
505
+ "grad_norm": 0.1557791829109192,
506
+ "learning_rate": 4.848221577856243e-05,
507
+ "loss": 1.1001962423324585,
508
+ "mean_token_accuracy": 0.6782753691077232,
509
+ "num_tokens": 4844482.0,
510
+ "step": 50
511
+ },
512
+ {
513
+ "entropy": 1.0782635807991028,
514
+ "epoch": 0.4939467312348668,
515
+ "grad_norm": 0.15607225894927979,
516
+ "learning_rate": 4.838985026440905e-05,
517
+ "loss": 1.0587979555130005,
518
+ "mean_token_accuracy": 0.6873368322849274,
519
+ "num_tokens": 4942340.0,
520
+ "step": 51
521
+ },
522
+ {
523
+ "entropy": 1.071622334420681,
524
+ "epoch": 0.5036319612590799,
525
+ "grad_norm": 0.1505449265241623,
526
+ "learning_rate": 4.829484999858815e-05,
527
+ "loss": 1.0695254802703857,
528
+ "mean_token_accuracy": 0.6835980787873268,
529
+ "num_tokens": 5040296.0,
530
+ "step": 52
531
+ },
532
+ {
533
+ "entropy": 1.0763383507728577,
534
+ "epoch": 0.513317191283293,
535
+ "grad_norm": 0.15331462025642395,
536
+ "learning_rate": 4.819722568241274e-05,
537
+ "loss": 1.073240041732788,
538
+ "mean_token_accuracy": 0.6868264526128769,
539
+ "num_tokens": 5136985.0,
540
+ "step": 53
541
+ },
542
+ {
543
+ "entropy": 1.086071863770485,
544
+ "epoch": 0.5230024213075061,
545
+ "grad_norm": 0.15281084179878235,
546
+ "learning_rate": 4.8096988312782174e-05,
547
+ "loss": 1.089070200920105,
548
+ "mean_token_accuracy": 0.6783855706453323,
549
+ "num_tokens": 5234555.0,
550
+ "step": 54
551
+ },
552
+ {
553
+ "entropy": 1.0728528648614883,
554
+ "epoch": 0.5326876513317191,
555
+ "grad_norm": 0.15144552290439606,
556
+ "learning_rate": 4.799414918094347e-05,
557
+ "loss": 1.0666192770004272,
558
+ "mean_token_accuracy": 0.6838826537132263,
559
+ "num_tokens": 5330627.0,
560
+ "step": 55
561
+ },
562
+ {
563
+ "entropy": 1.0615920722484589,
564
+ "epoch": 0.5423728813559322,
565
+ "grad_norm": 0.25972187519073486,
566
+ "learning_rate": 4.788871987121937e-05,
567
+ "loss": 1.0613664388656616,
568
+ "mean_token_accuracy": 0.6867948919534683,
569
+ "num_tokens": 5428196.0,
570
+ "step": 56
571
+ },
572
+ {
573
+ "entropy": 1.066960223019123,
574
+ "epoch": 0.5520581113801453,
575
+ "grad_norm": 0.15711794793605804,
576
+ "learning_rate": 4.77807122597034e-05,
577
+ "loss": 1.0630605220794678,
578
+ "mean_token_accuracy": 0.6852066963911057,
579
+ "num_tokens": 5525318.0,
580
+ "step": 57
581
+ },
582
+ {
583
+ "entropy": 1.0585280284285545,
584
+ "epoch": 0.5617433414043583,
585
+ "grad_norm": 0.154302716255188,
586
+ "learning_rate": 4.767013851292212e-05,
587
+ "loss": 1.048720359802246,
588
+ "mean_token_accuracy": 0.6897359937429428,
589
+ "num_tokens": 5623136.0,
590
+ "step": 58
591
+ },
592
+ {
593
+ "entropy": 1.0350576639175415,
594
+ "epoch": 0.5714285714285714,
595
+ "grad_norm": 0.15237243473529816,
596
+ "learning_rate": 4.755701108646463e-05,
597
+ "loss": 1.027522325515747,
598
+ "mean_token_accuracy": 0.6950520128011703,
599
+ "num_tokens": 5720541.0,
600
+ "step": 59
601
+ },
602
+ {
603
+ "entropy": 1.0580420941114426,
604
+ "epoch": 0.5811138014527845,
605
+ "grad_norm": 0.14978040754795074,
606
+ "learning_rate": 4.744134272357948e-05,
607
+ "loss": 1.0437177419662476,
608
+ "mean_token_accuracy": 0.6894178092479706,
609
+ "num_tokens": 5818052.0,
610
+ "step": 60
611
+ },
612
+ {
613
+ "entropy": 1.047768495976925,
614
+ "epoch": 0.5907990314769975,
615
+ "grad_norm": 0.45630910992622375,
616
+ "learning_rate": 4.732314645373921e-05,
617
+ "loss": 1.0360010862350464,
618
+ "mean_token_accuracy": 0.6936607509851456,
619
+ "num_tokens": 5915984.0,
620
+ "step": 61
621
+ },
622
+ {
623
+ "entropy": 1.0626960098743439,
624
+ "epoch": 0.6004842615012107,
625
+ "grad_norm": 0.2640538513660431,
626
+ "learning_rate": 4.7202435591172676e-05,
627
+ "loss": 1.0494108200073242,
628
+ "mean_token_accuracy": 0.6885051354765892,
629
+ "num_tokens": 6013024.0,
630
+ "step": 62
631
+ },
632
+ {
633
+ "entropy": 1.0739953070878983,
634
+ "epoch": 0.6101694915254238,
635
+ "grad_norm": 0.15643136203289032,
636
+ "learning_rate": 4.7079223733365237e-05,
637
+ "loss": 1.0664867162704468,
638
+ "mean_token_accuracy": 0.6844003722071648,
639
+ "num_tokens": 6110253.0,
640
+ "step": 63
641
+ },
642
+ {
643
+ "entropy": 1.0349700078368187,
644
+ "epoch": 0.6198547215496368,
645
+ "grad_norm": 0.1503216028213501,
646
+ "learning_rate": 4.6953524759527054e-05,
647
+ "loss": 1.0348119735717773,
648
+ "mean_token_accuracy": 0.692238837480545,
649
+ "num_tokens": 6207767.0,
650
+ "step": 64
651
+ },
652
+ {
653
+ "entropy": 1.034246288239956,
654
+ "epoch": 0.6295399515738499,
655
+ "grad_norm": 0.1513209342956543,
656
+ "learning_rate": 4.6825352829029705e-05,
657
+ "loss": 1.0328264236450195,
658
+ "mean_token_accuracy": 0.6916565299034119,
659
+ "num_tokens": 6305333.0,
660
+ "step": 65
661
+ },
662
+ {
663
+ "entropy": 1.0430887341499329,
664
+ "epoch": 0.639225181598063,
665
+ "grad_norm": 0.15611153841018677,
666
+ "learning_rate": 4.6694722379811185e-05,
667
+ "loss": 1.0414214134216309,
668
+ "mean_token_accuracy": 0.691117949783802,
669
+ "num_tokens": 6402785.0,
670
+ "step": 66
671
+ },
672
+ {
673
+ "entropy": 1.0569112747907639,
674
+ "epoch": 0.648910411622276,
675
+ "grad_norm": 0.1588103026151657,
676
+ "learning_rate": 4.656164812674951e-05,
677
+ "loss": 1.047569990158081,
678
+ "mean_token_accuracy": 0.6887474879622459,
679
+ "num_tokens": 6499919.0,
680
+ "step": 67
681
+ },
682
+ {
683
+ "entropy": 1.0378888696432114,
684
+ "epoch": 0.6585956416464891,
685
+ "grad_norm": 0.15132373571395874,
686
+ "learning_rate": 4.642614506000523e-05,
687
+ "loss": 1.0328857898712158,
688
+ "mean_token_accuracy": 0.6924383491277695,
689
+ "num_tokens": 6597204.0,
690
+ "step": 68
691
+ },
692
+ {
693
+ "entropy": 1.032283864915371,
694
+ "epoch": 0.6682808716707022,
695
+ "grad_norm": 0.1583699733018875,
696
+ "learning_rate": 4.628822844333278e-05,
697
+ "loss": 1.0151689052581787,
698
+ "mean_token_accuracy": 0.6986712738871574,
699
+ "num_tokens": 6694398.0,
700
+ "step": 69
701
+ },
702
+ {
703
+ "entropy": 1.0015813261270523,
704
+ "epoch": 0.6779661016949152,
705
+ "grad_norm": 0.1757071167230606,
706
+ "learning_rate": 4.614791381236115e-05,
707
+ "loss": 0.9920505285263062,
708
+ "mean_token_accuracy": 0.7040144950151443,
709
+ "num_tokens": 6791780.0,
710
+ "step": 70
711
+ },
712
+ {
713
+ "entropy": 1.0453337281942368,
714
+ "epoch": 0.6876513317191283,
715
+ "grad_norm": 0.1524701863527298,
716
+ "learning_rate": 4.6005216972843864e-05,
717
+ "loss": 1.0443840026855469,
718
+ "mean_token_accuracy": 0.6916699558496475,
719
+ "num_tokens": 6888953.0,
720
+ "step": 71
721
+ },
722
+ {
723
+ "entropy": 1.0230370834469795,
724
+ "epoch": 0.6973365617433414,
725
+ "grad_norm": 0.15993070602416992,
726
+ "learning_rate": 4.5860153998878494e-05,
727
+ "loss": 1.0304991006851196,
728
+ "mean_token_accuracy": 0.6937867403030396,
729
+ "num_tokens": 6986471.0,
730
+ "step": 72
731
+ },
732
+ {
733
+ "entropy": 1.0421988740563393,
734
+ "epoch": 0.7070217917675545,
735
+ "grad_norm": 0.1573047935962677,
736
+ "learning_rate": 4.571274123109606e-05,
737
+ "loss": 1.0444982051849365,
738
+ "mean_token_accuracy": 0.6912301033735275,
739
+ "num_tokens": 7083829.0,
740
+ "step": 73
741
+ },
742
+ {
743
+ "entropy": 1.0748334154486656,
744
+ "epoch": 0.7167070217917676,
745
+ "grad_norm": 0.16008728742599487,
746
+ "learning_rate": 4.5562995274820284e-05,
747
+ "loss": 1.0726516246795654,
748
+ "mean_token_accuracy": 0.6823427230119705,
749
+ "num_tokens": 7180818.0,
750
+ "step": 74
751
+ },
752
+ {
753
+ "entropy": 1.0194222182035446,
754
+ "epoch": 0.7263922518159807,
755
+ "grad_norm": 0.16299541294574738,
756
+ "learning_rate": 4.541093299819714e-05,
757
+ "loss": 1.003331184387207,
758
+ "mean_token_accuracy": 0.6996421813964844,
759
+ "num_tokens": 7277648.0,
760
+ "step": 75
761
+ },
762
+ {
763
+ "entropy": 1.0004182010889053,
764
+ "epoch": 0.7360774818401937,
765
+ "grad_norm": 0.15372581779956818,
766
+ "learning_rate": 4.5256571530294664e-05,
767
+ "loss": 0.9760631918907166,
768
+ "mean_token_accuracy": 0.7064241617918015,
769
+ "num_tokens": 7375160.0,
770
+ "step": 76
771
+ },
772
+ {
773
+ "entropy": 1.0313043221831322,
774
+ "epoch": 0.7457627118644068,
775
+ "grad_norm": 0.15582375228405,
776
+ "learning_rate": 4.5099928259173516e-05,
777
+ "loss": 1.0199785232543945,
778
+ "mean_token_accuracy": 0.6969870626926422,
779
+ "num_tokens": 7472481.0,
780
+ "step": 77
781
+ },
782
+ {
783
+ "entropy": 1.0267394110560417,
784
+ "epoch": 0.7554479418886199,
785
+ "grad_norm": 0.15595640242099762,
786
+ "learning_rate": 4.4941020829928245e-05,
787
+ "loss": 1.0236713886260986,
788
+ "mean_token_accuracy": 0.697633869946003,
789
+ "num_tokens": 7569978.0,
790
+ "step": 78
791
+ },
792
+ {
793
+ "entropy": 0.9967087134718895,
794
+ "epoch": 0.7651331719128329,
795
+ "grad_norm": 0.15426741540431976,
796
+ "learning_rate": 4.477986714269972e-05,
797
+ "loss": 0.9990708231925964,
798
+ "mean_token_accuracy": 0.7013676688075066,
799
+ "num_tokens": 7668053.0,
800
+ "step": 79
801
+ },
802
+ {
803
+ "entropy": 0.9936242178082466,
804
+ "epoch": 0.774818401937046,
805
+ "grad_norm": 0.15127019584178925,
806
+ "learning_rate": 4.4616485350658685e-05,
807
+ "loss": 0.992855429649353,
808
+ "mean_token_accuracy": 0.7030186578631401,
809
+ "num_tokens": 7764243.0,
810
+ "step": 80
811
+ },
812
+ {
813
+ "entropy": 1.0158754512667656,
814
+ "epoch": 0.784503631961259,
815
+ "grad_norm": 0.1566711664199829,
816
+ "learning_rate": 4.445089385796099e-05,
817
+ "loss": 1.0183484554290771,
818
+ "mean_token_accuracy": 0.6949350982904434,
819
+ "num_tokens": 7860199.0,
820
+ "step": 81
821
+ },
822
+ {
823
+ "entropy": 1.035573497414589,
824
+ "epoch": 0.7941888619854721,
825
+ "grad_norm": 0.15846672654151917,
826
+ "learning_rate": 4.4283111317674374e-05,
827
+ "loss": 1.0358930826187134,
828
+ "mean_token_accuracy": 0.6919568032026291,
829
+ "num_tokens": 7956920.0,
830
+ "step": 82
831
+ },
832
+ {
833
+ "entropy": 1.0298198834061623,
834
+ "epoch": 0.8038740920096852,
835
+ "grad_norm": 0.1551441252231598,
836
+ "learning_rate": 4.4113156629677316e-05,
837
+ "loss": 1.02339768409729,
838
+ "mean_token_accuracy": 0.6937244087457657,
839
+ "num_tokens": 8053766.0,
840
+ "step": 83
841
+ },
842
+ {
843
+ "entropy": 1.0543159544467926,
844
+ "epoch": 0.8135593220338984,
845
+ "grad_norm": 0.15082016587257385,
846
+ "learning_rate": 4.394104893853007e-05,
847
+ "loss": 1.0417697429656982,
848
+ "mean_token_accuracy": 0.6908006444573402,
849
+ "num_tokens": 8151263.0,
850
+ "step": 84
851
+ },
852
+ {
853
+ "entropy": 0.9868235737085342,
854
+ "epoch": 0.8232445520581114,
855
+ "grad_norm": 0.15088169276714325,
856
+ "learning_rate": 4.3766807631318106e-05,
857
+ "loss": 0.9730858206748962,
858
+ "mean_token_accuracy": 0.7068010121583939,
859
+ "num_tokens": 8249031.0,
860
+ "step": 85
861
+ },
862
+ {
863
+ "entropy": 1.0439471304416656,
864
+ "epoch": 0.8329297820823245,
865
+ "grad_norm": 0.15690501034259796,
866
+ "learning_rate": 4.359045233546827e-05,
867
+ "loss": 1.033866286277771,
868
+ "mean_token_accuracy": 0.6926037818193436,
869
+ "num_tokens": 8345911.0,
870
+ "step": 86
871
+ },
872
+ {
873
+ "entropy": 1.0188320130109787,
874
+ "epoch": 0.8426150121065376,
875
+ "grad_norm": 0.16062293946743011,
876
+ "learning_rate": 4.341200291653781e-05,
877
+ "loss": 1.0072400569915771,
878
+ "mean_token_accuracy": 0.7004816979169846,
879
+ "num_tokens": 8440245.0,
880
+ "step": 87
881
+ },
882
+ {
883
+ "entropy": 1.0310387164354324,
884
+ "epoch": 0.8523002421307506,
885
+ "grad_norm": 0.15309439599514008,
886
+ "learning_rate": 4.323147947597666e-05,
887
+ "loss": 1.0136595964431763,
888
+ "mean_token_accuracy": 0.6977442428469658,
889
+ "num_tokens": 8537604.0,
890
+ "step": 88
891
+ },
892
+ {
893
+ "entropy": 1.0147035270929337,
894
+ "epoch": 0.8619854721549637,
895
+ "grad_norm": 0.16039453446865082,
896
+ "learning_rate": 4.3048902348863116e-05,
897
+ "loss": 1.0053828954696655,
898
+ "mean_token_accuracy": 0.6996104419231415,
899
+ "num_tokens": 8634832.0,
900
+ "step": 89
901
+ },
902
+ {
903
+ "entropy": 1.009638950228691,
904
+ "epoch": 0.8716707021791767,
905
+ "grad_norm": 0.15286219120025635,
906
+ "learning_rate": 4.2864292101613136e-05,
907
+ "loss": 1.012491226196289,
908
+ "mean_token_accuracy": 0.6957853958010674,
909
+ "num_tokens": 8732321.0,
910
+ "step": 90
911
+ },
912
+ {
913
+ "entropy": 1.0169770643115044,
914
+ "epoch": 0.8813559322033898,
915
+ "grad_norm": 0.16297462582588196,
916
+ "learning_rate": 4.267766952966369e-05,
917
+ "loss": 1.0225484371185303,
918
+ "mean_token_accuracy": 0.6936622560024261,
919
+ "num_tokens": 8830140.0,
920
+ "step": 91
921
+ },
922
+ {
923
+ "entropy": 0.9886957630515099,
924
+ "epoch": 0.8910411622276029,
925
+ "grad_norm": 0.18117401003837585,
926
+ "learning_rate": 4.248905565513023e-05,
927
+ "loss": 0.9961055517196655,
928
+ "mean_token_accuracy": 0.699328251183033,
929
+ "num_tokens": 8926792.0,
930
+ "step": 92
931
+ },
932
+ {
933
+ "entropy": 1.026213452219963,
934
+ "epoch": 0.9007263922518159,
935
+ "grad_norm": 0.15571565926074982,
936
+ "learning_rate": 4.229847172443866e-05,
937
+ "loss": 1.0185178518295288,
938
+ "mean_token_accuracy": 0.6954792588949203,
939
+ "num_tokens": 9024094.0,
940
+ "step": 93
941
+ },
942
+ {
943
+ "entropy": 0.9805751740932465,
944
+ "epoch": 0.910411622276029,
945
+ "grad_norm": 0.1524103581905365,
946
+ "learning_rate": 4.210593920593201e-05,
947
+ "loss": 0.9706284999847412,
948
+ "mean_token_accuracy": 0.7079346030950546,
949
+ "num_tokens": 9120978.0,
950
+ "step": 94
951
+ },
952
+ {
953
+ "entropy": 1.0263385400176048,
954
+ "epoch": 0.9200968523002422,
955
+ "grad_norm": 0.1659349501132965,
956
+ "learning_rate": 4.191147978745218e-05,
957
+ "loss": 1.010141134262085,
958
+ "mean_token_accuracy": 0.6972236707806587,
959
+ "num_tokens": 9217940.0,
960
+ "step": 95
961
+ },
962
+ {
963
+ "entropy": 0.9710419476032257,
964
+ "epoch": 0.9297820823244553,
965
+ "grad_norm": 0.14841604232788086,
966
+ "learning_rate": 4.171511537389684e-05,
967
+ "loss": 0.9553003311157227,
968
+ "mean_token_accuracy": 0.7146916389465332,
969
+ "num_tokens": 9315038.0,
970
+ "step": 96
971
+ },
972
+ {
973
+ "entropy": 1.0402830019593239,
974
+ "epoch": 0.9394673123486683,
975
+ "grad_norm": 0.1693028062582016,
976
+ "learning_rate": 4.151686808475204e-05,
977
+ "loss": 1.022579312324524,
978
+ "mean_token_accuracy": 0.6941675096750259,
979
+ "num_tokens": 9411796.0,
980
+ "step": 97
981
+ },
982
+ {
983
+ "entropy": 0.9890827238559723,
984
+ "epoch": 0.9491525423728814,
985
+ "grad_norm": 0.15846097469329834,
986
+ "learning_rate": 4.131676025160047e-05,
987
+ "loss": 0.9882045984268188,
988
+ "mean_token_accuracy": 0.7028019949793816,
989
+ "num_tokens": 9508011.0,
990
+ "step": 98
991
+ },
992
+ {
993
+ "entropy": 0.964131772518158,
994
+ "epoch": 0.9588377723970944,
995
+ "grad_norm": 0.1603643000125885,
996
+ "learning_rate": 4.111481441560598e-05,
997
+ "loss": 0.9728220701217651,
998
+ "mean_token_accuracy": 0.7061599791049957,
999
+ "num_tokens": 9605557.0,
1000
+ "step": 99
1001
+ },
1002
+ {
1003
+ "entropy": 1.0129969343543053,
1004
+ "epoch": 0.9685230024213075,
1005
+ "grad_norm": 0.1754368096590042,
1006
+ "learning_rate": 4.091105332497439e-05,
1007
+ "loss": 1.0291517972946167,
1008
+ "mean_token_accuracy": 0.6916319876909256,
1009
+ "num_tokens": 9703059.0,
1010
+ "step": 100
1011
+ },
1012
+ {
1013
+ "entropy": 1.0249989926815033,
1014
+ "epoch": 0.9782082324455206,
1015
+ "grad_norm": 0.16457504034042358,
1016
+ "learning_rate": 4.070549993239106e-05,
1017
+ "loss": 1.0286558866500854,
1018
+ "mean_token_accuracy": 0.6931352093815804,
1019
+ "num_tokens": 9800187.0,
1020
+ "step": 101
1021
+ },
1022
+ {
1023
+ "entropy": 1.0157797187566757,
1024
+ "epoch": 0.9878934624697336,
1025
+ "grad_norm": 0.1640177220106125,
1026
+ "learning_rate": 4.049817739243532e-05,
1027
+ "loss": 1.001893162727356,
1028
+ "mean_token_accuracy": 0.6987328082323074,
1029
+ "num_tokens": 9897311.0,
1030
+ "step": 102
1031
+ },
1032
+ {
1033
+ "entropy": 1.0060301199555397,
1034
+ "epoch": 0.9975786924939467,
1035
+ "grad_norm": 0.15887762606143951,
1036
+ "learning_rate": 4.028910905897229e-05,
1037
+ "loss": 0.9999165534973145,
1038
+ "mean_token_accuracy": 0.7007258981466293,
1039
+ "num_tokens": 9994136.0,
1040
+ "step": 103
1041
+ },
1042
+ {
1043
+ "entropy": 1.057703673839569,
1044
+ "epoch": 1.0,
1045
+ "grad_norm": 0.2889890670776367,
1046
+ "learning_rate": 4.007831848252211e-05,
1047
+ "loss": 1.043251395225525,
1048
+ "mean_token_accuracy": 0.6889578104019165,
1049
+ "num_tokens": 10018679.0,
1050
+ "step": 104
1051
+ },
1052
+ {
1053
+ "entropy": 1.0108226910233498,
1054
+ "epoch": 1.0096852300242132,
1055
+ "grad_norm": 0.1708330363035202,
1056
+ "learning_rate": 3.986582940760717e-05,
1057
+ "loss": 0.9764024615287781,
1058
+ "mean_token_accuracy": 0.7037361264228821,
1059
+ "num_tokens": 10115941.0,
1060
+ "step": 105
1061
+ },
1062
+ {
1063
+ "entropy": 1.002457171678543,
1064
+ "epoch": 1.0193704600484261,
1065
+ "grad_norm": 0.16839711368083954,
1066
+ "learning_rate": 3.965166577007733e-05,
1067
+ "loss": 0.9664080142974854,
1068
+ "mean_token_accuracy": 0.7054638564586639,
1069
+ "num_tokens": 10212419.0,
1070
+ "step": 106
1071
+ },
1072
+ {
1073
+ "entropy": 0.955726683139801,
1074
+ "epoch": 1.0290556900726393,
1075
+ "grad_norm": 0.1585574597120285,
1076
+ "learning_rate": 3.94358516944137e-05,
1077
+ "loss": 0.9458193778991699,
1078
+ "mean_token_accuracy": 0.7124402225017548,
1079
+ "num_tokens": 10309395.0,
1080
+ "step": 107
1081
+ },
1082
+ {
1083
+ "entropy": 0.9474495872855186,
1084
+ "epoch": 1.0387409200968523,
1085
+ "grad_norm": 0.17470364272594452,
1086
+ "learning_rate": 3.9218411491011176e-05,
1087
+ "loss": 0.9364479780197144,
1088
+ "mean_token_accuracy": 0.7168606445193291,
1089
+ "num_tokens": 10407031.0,
1090
+ "step": 108
1091
+ },
1092
+ {
1093
+ "entropy": 0.9753162637352943,
1094
+ "epoch": 1.0484261501210654,
1095
+ "grad_norm": 0.17014382779598236,
1096
+ "learning_rate": 3.899936965343989e-05,
1097
+ "loss": 0.9700603485107422,
1098
+ "mean_token_accuracy": 0.705869048833847,
1099
+ "num_tokens": 10504190.0,
1100
+ "step": 109
1101
+ },
1102
+ {
1103
+ "entropy": 0.9242235645651817,
1104
+ "epoch": 1.0581113801452784,
1105
+ "grad_norm": 0.16195626556873322,
1106
+ "learning_rate": 3.877875085568623e-05,
1107
+ "loss": 0.9271841049194336,
1108
+ "mean_token_accuracy": 0.715888999402523,
1109
+ "num_tokens": 10601162.0,
1110
+ "step": 110
1111
+ },
1112
+ {
1113
+ "entropy": 0.9163908809423447,
1114
+ "epoch": 1.0677966101694916,
1115
+ "grad_norm": 0.16627757251262665,
1116
+ "learning_rate": 3.8556579949373384e-05,
1117
+ "loss": 0.9301415681838989,
1118
+ "mean_token_accuracy": 0.7159750387072563,
1119
+ "num_tokens": 10697647.0,
1120
+ "step": 111
1121
+ },
1122
+ {
1123
+ "entropy": 0.9796807840466499,
1124
+ "epoch": 1.0774818401937045,
1125
+ "grad_norm": 0.16515608131885529,
1126
+ "learning_rate": 3.833288196096194e-05,
1127
+ "loss": 0.9877065420150757,
1128
+ "mean_token_accuracy": 0.7009606212377548,
1129
+ "num_tokens": 10795421.0,
1130
+ "step": 112
1131
+ },
1132
+ {
1133
+ "entropy": 0.9254954382777214,
1134
+ "epoch": 1.0871670702179177,
1135
+ "grad_norm": 0.17847532033920288,
1136
+ "learning_rate": 3.8107682088930794e-05,
1137
+ "loss": 0.9220598936080933,
1138
+ "mean_token_accuracy": 0.7181012779474258,
1139
+ "num_tokens": 10889471.0,
1140
+ "step": 113
1141
+ },
1142
+ {
1143
+ "entropy": 0.9571070671081543,
1144
+ "epoch": 1.0968523002421307,
1145
+ "grad_norm": 0.16064082086086273,
1146
+ "learning_rate": 3.7881005700938635e-05,
1147
+ "loss": 0.9418208003044128,
1148
+ "mean_token_accuracy": 0.7128180488944054,
1149
+ "num_tokens": 10986793.0,
1150
+ "step": 114
1151
+ },
1152
+ {
1153
+ "entropy": 0.9537178128957748,
1154
+ "epoch": 1.1065375302663438,
1155
+ "grad_norm": 0.16848786175251007,
1156
+ "learning_rate": 3.7652878330966415e-05,
1157
+ "loss": 0.938056468963623,
1158
+ "mean_token_accuracy": 0.7114301025867462,
1159
+ "num_tokens": 11083742.0,
1160
+ "step": 115
1161
+ },
1162
+ {
1163
+ "entropy": 0.9795895889401436,
1164
+ "epoch": 1.116222760290557,
1165
+ "grad_norm": 0.16908369958400726,
1166
+ "learning_rate": 3.742332567644107e-05,
1167
+ "loss": 0.9716662168502808,
1168
+ "mean_token_accuracy": 0.703265480697155,
1169
+ "num_tokens": 11181520.0,
1170
+ "step": 116
1171
+ },
1172
+ {
1173
+ "entropy": 0.9542344212532043,
1174
+ "epoch": 1.12590799031477,
1175
+ "grad_norm": 0.16320371627807617,
1176
+ "learning_rate": 3.719237359534087e-05,
1177
+ "loss": 0.94194495677948,
1178
+ "mean_token_accuracy": 0.7138355523347855,
1179
+ "num_tokens": 11279085.0,
1180
+ "step": 117
1181
+ },
1182
+ {
1183
+ "entropy": 0.9827202260494232,
1184
+ "epoch": 1.1355932203389831,
1185
+ "grad_norm": 0.1600467562675476,
1186
+ "learning_rate": 3.6960048103282566e-05,
1187
+ "loss": 0.9698864817619324,
1188
+ "mean_token_accuracy": 0.7054214552044868,
1189
+ "num_tokens": 11375898.0,
1190
+ "step": 118
1191
+ },
1192
+ {
1193
+ "entropy": 0.953464925289154,
1194
+ "epoch": 1.145278450363196,
1195
+ "grad_norm": 0.16535067558288574,
1196
+ "learning_rate": 3.672637537059093e-05,
1197
+ "loss": 0.9407245516777039,
1198
+ "mean_token_accuracy": 0.7140245661139488,
1199
+ "num_tokens": 11473117.0,
1200
+ "step": 119
1201
+ },
1202
+ {
1203
+ "entropy": 0.9346380978822708,
1204
+ "epoch": 1.1549636803874093,
1205
+ "grad_norm": 0.16306935250759125,
1206
+ "learning_rate": 3.6491381719350756e-05,
1207
+ "loss": 0.9232890009880066,
1208
+ "mean_token_accuracy": 0.7176763042807579,
1209
+ "num_tokens": 11569815.0,
1210
+ "step": 120
1211
+ },
1212
+ {
1213
+ "entropy": 0.9564549624919891,
1214
+ "epoch": 1.1646489104116222,
1215
+ "grad_norm": 0.17123237252235413,
1216
+ "learning_rate": 3.6255093620441834e-05,
1217
+ "loss": 0.9536066055297852,
1218
+ "mean_token_accuracy": 0.7104167863726616,
1219
+ "num_tokens": 11667629.0,
1220
+ "step": 121
1221
+ },
1222
+ {
1223
+ "entropy": 0.9488897398114204,
1224
+ "epoch": 1.1743341404358354,
1225
+ "grad_norm": 0.1677977293729782,
1226
+ "learning_rate": 3.6017537690557115e-05,
1227
+ "loss": 0.9455256462097168,
1228
+ "mean_token_accuracy": 0.7133829072117805,
1229
+ "num_tokens": 11764727.0,
1230
+ "step": 122
1231
+ },
1232
+ {
1233
+ "entropy": 0.9661734327673912,
1234
+ "epoch": 1.1840193704600483,
1235
+ "grad_norm": 0.16724446415901184,
1236
+ "learning_rate": 3.577874068920446e-05,
1237
+ "loss": 0.9509412050247192,
1238
+ "mean_token_accuracy": 0.7083179950714111,
1239
+ "num_tokens": 11861449.0,
1240
+ "step": 123
1241
+ },
1242
+ {
1243
+ "entropy": 0.9373920634388924,
1244
+ "epoch": 1.1937046004842615,
1245
+ "grad_norm": 0.1661859154701233,
1246
+ "learning_rate": 3.553872951569236e-05,
1247
+ "loss": 0.9326287508010864,
1248
+ "mean_token_accuracy": 0.7153337746858597,
1249
+ "num_tokens": 11957739.0,
1250
+ "step": 124
1251
+ },
1252
+ {
1253
+ "entropy": 0.9740064963698387,
1254
+ "epoch": 1.2033898305084745,
1255
+ "grad_norm": 0.1778748631477356,
1256
+ "learning_rate": 3.529753120609982e-05,
1257
+ "loss": 0.9758471846580505,
1258
+ "mean_token_accuracy": 0.7040262669324875,
1259
+ "num_tokens": 12055929.0,
1260
+ "step": 125
1261
+ },
1262
+ {
1263
+ "entropy": 0.9446261376142502,
1264
+ "epoch": 1.2130750605326877,
1265
+ "grad_norm": 0.1627563238143921,
1266
+ "learning_rate": 3.505517293023088e-05,
1267
+ "loss": 0.9395431280136108,
1268
+ "mean_token_accuracy": 0.7127633616328239,
1269
+ "num_tokens": 12153914.0,
1270
+ "step": 126
1271
+ },
1272
+ {
1273
+ "entropy": 0.9552619904279709,
1274
+ "epoch": 1.2227602905569008,
1275
+ "grad_norm": 0.16823652386665344,
1276
+ "learning_rate": 3.481168198855409e-05,
1277
+ "loss": 0.9568450450897217,
1278
+ "mean_token_accuracy": 0.7067705914378166,
1279
+ "num_tokens": 12250252.0,
1280
+ "step": 127
1281
+ },
1282
+ {
1283
+ "entropy": 0.9257349297404289,
1284
+ "epoch": 1.2324455205811138,
1285
+ "grad_norm": 0.1990891546010971,
1286
+ "learning_rate": 3.456708580912725e-05,
1287
+ "loss": 0.917294442653656,
1288
+ "mean_token_accuracy": 0.7171075865626335,
1289
+ "num_tokens": 12347749.0,
1290
+ "step": 128
1291
+ },
1292
+ {
1293
+ "entropy": 0.9262085407972336,
1294
+ "epoch": 1.242130750605327,
1295
+ "grad_norm": 0.16901157796382904,
1296
+ "learning_rate": 3.432141194450772e-05,
1297
+ "loss": 0.9202949404716492,
1298
+ "mean_token_accuracy": 0.7190263494849205,
1299
+ "num_tokens": 12445530.0,
1300
+ "step": 129
1301
+ },
1302
+ {
1303
+ "entropy": 0.9542277082800865,
1304
+ "epoch": 1.25181598062954,
1305
+ "grad_norm": 0.16794957220554352,
1306
+ "learning_rate": 3.407468806864883e-05,
1307
+ "loss": 0.9502478837966919,
1308
+ "mean_token_accuracy": 0.7078473269939423,
1309
+ "num_tokens": 12542439.0,
1310
+ "step": 130
1311
+ },
1312
+ {
1313
+ "entropy": 0.9414918050169945,
1314
+ "epoch": 1.261501210653753,
1315
+ "grad_norm": 0.1646578013896942,
1316
+ "learning_rate": 3.382694197378252e-05,
1317
+ "loss": 0.9400969743728638,
1318
+ "mean_token_accuracy": 0.7122456803917885,
1319
+ "num_tokens": 12639236.0,
1320
+ "step": 131
1321
+ },
1322
+ {
1323
+ "entropy": 0.9028183594346046,
1324
+ "epoch": 1.271186440677966,
1325
+ "grad_norm": 0.16290493309497833,
1326
+ "learning_rate": 3.357820156728867e-05,
1327
+ "loss": 0.8925794959068298,
1328
+ "mean_token_accuracy": 0.7234794571995735,
1329
+ "num_tokens": 12735541.0,
1330
+ "step": 132
1331
+ },
1332
+ {
1333
+ "entropy": 0.9390545934438705,
1334
+ "epoch": 1.2808716707021792,
1335
+ "grad_norm": 0.16618622839450836,
1336
+ "learning_rate": 3.332849486855144e-05,
1337
+ "loss": 0.9257231950759888,
1338
+ "mean_token_accuracy": 0.7156568765640259,
1339
+ "num_tokens": 12831913.0,
1340
+ "step": 133
1341
+ },
1342
+ {
1343
+ "entropy": 0.9283222854137421,
1344
+ "epoch": 1.2905569007263922,
1345
+ "grad_norm": 0.1611809879541397,
1346
+ "learning_rate": 3.307785000580313e-05,
1347
+ "loss": 0.9223712682723999,
1348
+ "mean_token_accuracy": 0.718876950442791,
1349
+ "num_tokens": 12927846.0,
1350
+ "step": 134
1351
+ },
1352
+ {
1353
+ "entropy": 0.9606467336416245,
1354
+ "epoch": 1.3002421307506054,
1355
+ "grad_norm": 0.16902242600917816,
1356
+ "learning_rate": 3.282629521295556e-05,
1357
+ "loss": 0.9587893486022949,
1358
+ "mean_token_accuracy": 0.7081739977002144,
1359
+ "num_tokens": 13025237.0,
1360
+ "step": 135
1361
+ },
1362
+ {
1363
+ "entropy": 0.9471021741628647,
1364
+ "epoch": 1.3099273607748185,
1365
+ "grad_norm": 0.1694117784500122,
1366
+ "learning_rate": 3.257385882641971e-05,
1367
+ "loss": 0.9391512870788574,
1368
+ "mean_token_accuracy": 0.7138297259807587,
1369
+ "num_tokens": 13122245.0,
1370
+ "step": 136
1371
+ },
1372
+ {
1373
+ "entropy": 0.9396388903260231,
1374
+ "epoch": 1.3196125907990315,
1375
+ "grad_norm": 0.16647525131702423,
1376
+ "learning_rate": 3.232056928191376e-05,
1377
+ "loss": 0.922247052192688,
1378
+ "mean_token_accuracy": 0.7196550145745277,
1379
+ "num_tokens": 13219660.0,
1380
+ "step": 137
1381
+ },
1382
+ {
1383
+ "entropy": 0.9275924190878868,
1384
+ "epoch": 1.3292978208232444,
1385
+ "grad_norm": 0.163606196641922,
1386
+ "learning_rate": 3.2066455111259945e-05,
1387
+ "loss": 0.9154177904129028,
1388
+ "mean_token_accuracy": 0.7186660617589951,
1389
+ "num_tokens": 13316979.0,
1390
+ "step": 138
1391
+ },
1392
+ {
1393
+ "entropy": 0.9340094104409218,
1394
+ "epoch": 1.3389830508474576,
1395
+ "grad_norm": 0.17434757947921753,
1396
+ "learning_rate": 3.1811544939170575e-05,
1397
+ "loss": 0.9282485842704773,
1398
+ "mean_token_accuracy": 0.7156421467661858,
1399
+ "num_tokens": 13413652.0,
1400
+ "step": 139
1401
+ },
1402
+ {
1403
+ "entropy": 0.9297875389456749,
1404
+ "epoch": 1.3486682808716708,
1405
+ "grad_norm": 0.1651269793510437,
1406
+ "learning_rate": 3.155586748002362e-05,
1407
+ "loss": 0.9184130430221558,
1408
+ "mean_token_accuracy": 0.7187451273202896,
1409
+ "num_tokens": 13510441.0,
1410
+ "step": 140
1411
+ },
1412
+ {
1413
+ "entropy": 0.920344740152359,
1414
+ "epoch": 1.3583535108958837,
1415
+ "grad_norm": 0.18017897009849548,
1416
+ "learning_rate": 3.1299451534628135e-05,
1417
+ "loss": 0.9237980842590332,
1418
+ "mean_token_accuracy": 0.7185762003064156,
1419
+ "num_tokens": 13607138.0,
1420
+ "step": 141
1421
+ },
1422
+ {
1423
+ "entropy": 0.9063275828957558,
1424
+ "epoch": 1.368038740920097,
1425
+ "grad_norm": 0.1606295108795166,
1426
+ "learning_rate": 3.1042325986980064e-05,
1427
+ "loss": 0.9120801687240601,
1428
+ "mean_token_accuracy": 0.7203933596611023,
1429
+ "num_tokens": 13703849.0,
1430
+ "step": 142
1431
+ },
1432
+ {
1433
+ "entropy": 0.9234895408153534,
1434
+ "epoch": 1.3777239709443099,
1435
+ "grad_norm": 0.16578350961208344,
1436
+ "learning_rate": 3.0784519801008546e-05,
1437
+ "loss": 0.9238073825836182,
1438
+ "mean_token_accuracy": 0.7186264097690582,
1439
+ "num_tokens": 13801402.0,
1440
+ "step": 143
1441
+ },
1442
+ {
1443
+ "entropy": 0.9647551253437996,
1444
+ "epoch": 1.387409200968523,
1445
+ "grad_norm": 0.16832281649112701,
1446
+ "learning_rate": 3.0526062017313254e-05,
1447
+ "loss": 0.9546170234680176,
1448
+ "mean_token_accuracy": 0.7065940275788307,
1449
+ "num_tokens": 13898860.0,
1450
+ "step": 144
1451
+ },
1452
+ {
1453
+ "entropy": 0.9435562416911125,
1454
+ "epoch": 1.397094430992736,
1455
+ "grad_norm": 0.16285553574562073,
1456
+ "learning_rate": 3.0266981749893157e-05,
1457
+ "loss": 0.9355381727218628,
1458
+ "mean_token_accuracy": 0.7139561623334885,
1459
+ "num_tokens": 13996224.0,
1460
+ "step": 145
1461
+ },
1462
+ {
1463
+ "entropy": 0.9463768005371094,
1464
+ "epoch": 1.4067796610169492,
1465
+ "grad_norm": 0.18000875413417816,
1466
+ "learning_rate": 3.0007308182866985e-05,
1467
+ "loss": 0.9321545958518982,
1468
+ "mean_token_accuracy": 0.7152583748102188,
1469
+ "num_tokens": 14092953.0,
1470
+ "step": 146
1471
+ },
1472
+ {
1473
+ "entropy": 0.9378436356782913,
1474
+ "epoch": 1.4164648910411621,
1475
+ "grad_norm": 0.3399038016796112,
1476
+ "learning_rate": 2.974707056718571e-05,
1477
+ "loss": 0.9353235960006714,
1478
+ "mean_token_accuracy": 0.7144726887345314,
1479
+ "num_tokens": 14188222.0,
1480
+ "step": 147
1481
+ },
1482
+ {
1483
+ "entropy": 0.926091730594635,
1484
+ "epoch": 1.4261501210653753,
1485
+ "grad_norm": 0.1794121265411377,
1486
+ "learning_rate": 2.9486298217337637e-05,
1487
+ "loss": 0.927035927772522,
1488
+ "mean_token_accuracy": 0.7152861431241035,
1489
+ "num_tokens": 14284661.0,
1490
+ "step": 148
1491
+ },
1492
+ {
1493
+ "entropy": 0.9360547363758087,
1494
+ "epoch": 1.4358353510895885,
1495
+ "grad_norm": 0.16639335453510284,
1496
+ "learning_rate": 2.9225020508046232e-05,
1497
+ "loss": 0.9402387142181396,
1498
+ "mean_token_accuracy": 0.7152109816670418,
1499
+ "num_tokens": 14381541.0,
1500
+ "step": 149
1501
+ },
1502
+ {
1503
+ "entropy": 0.9604831784963608,
1504
+ "epoch": 1.4455205811138014,
1505
+ "grad_norm": 0.16612860560417175,
1506
+ "learning_rate": 2.8963266870961226e-05,
1507
+ "loss": 0.9555082321166992,
1508
+ "mean_token_accuracy": 0.7080897018313408,
1509
+ "num_tokens": 14479362.0,
1510
+ "step": 150
1511
+ },
1512
+ {
1513
+ "entropy": 0.9597497507929802,
1514
+ "epoch": 1.4552058111380144,
1515
+ "grad_norm": 0.1659996509552002,
1516
+ "learning_rate": 2.8701066791343288e-05,
1517
+ "loss": 0.9564999341964722,
1518
+ "mean_token_accuracy": 0.7071296945214272,
1519
+ "num_tokens": 14576735.0,
1520
+ "step": 151
1521
+ },
1522
+ {
1523
+ "entropy": 0.9736971110105515,
1524
+ "epoch": 1.4648910411622276,
1525
+ "grad_norm": 0.17209328711032867,
1526
+ "learning_rate": 2.8438449804742628e-05,
1527
+ "loss": 0.9558152556419373,
1528
+ "mean_token_accuracy": 0.7066713944077492,
1529
+ "num_tokens": 14674426.0,
1530
+ "step": 152
1531
+ },
1532
+ {
1533
+ "entropy": 0.9413225129246712,
1534
+ "epoch": 1.4745762711864407,
1535
+ "grad_norm": 0.16857822239398956,
1536
+ "learning_rate": 2.8175445493671972e-05,
1537
+ "loss": 0.9316411018371582,
1538
+ "mean_token_accuracy": 0.7146977111697197,
1539
+ "num_tokens": 14771566.0,
1540
+ "step": 153
1541
+ },
1542
+ {
1543
+ "entropy": 0.9432979747653008,
1544
+ "epoch": 1.4842615012106537,
1545
+ "grad_norm": 0.1611289530992508,
1546
+ "learning_rate": 2.7912083484274266e-05,
1547
+ "loss": 0.93993079662323,
1548
+ "mean_token_accuracy": 0.7159048616886139,
1549
+ "num_tokens": 14868581.0,
1550
+ "step": 154
1551
+ },
1552
+ {
1553
+ "entropy": 0.9212874621152878,
1554
+ "epoch": 1.4939467312348669,
1555
+ "grad_norm": 0.16538380086421967,
1556
+ "learning_rate": 2.7648393442985403e-05,
1557
+ "loss": 0.9123656749725342,
1558
+ "mean_token_accuracy": 0.719389833509922,
1559
+ "num_tokens": 14966068.0,
1560
+ "step": 155
1561
+ },
1562
+ {
1563
+ "entropy": 0.9139806106686592,
1564
+ "epoch": 1.5036319612590798,
1565
+ "grad_norm": 0.16073596477508545,
1566
+ "learning_rate": 2.7384405073192454e-05,
1567
+ "loss": 0.9082322120666504,
1568
+ "mean_token_accuracy": 0.7201628461480141,
1569
+ "num_tokens": 15063317.0,
1570
+ "step": 156
1571
+ },
1572
+ {
1573
+ "entropy": 0.9522331357002258,
1574
+ "epoch": 1.513317191283293,
1575
+ "grad_norm": 0.16658230125904083,
1576
+ "learning_rate": 2.7120148111887732e-05,
1577
+ "loss": 0.944631040096283,
1578
+ "mean_token_accuracy": 0.7112195938825607,
1579
+ "num_tokens": 15156906.0,
1580
+ "step": 157
1581
+ },
1582
+ {
1583
+ "entropy": 0.9170691967010498,
1584
+ "epoch": 1.5230024213075062,
1585
+ "grad_norm": 0.16504573822021484,
1586
+ "learning_rate": 2.6855652326319063e-05,
1587
+ "loss": 0.9089013934135437,
1588
+ "mean_token_accuracy": 0.7200009673833847,
1589
+ "num_tokens": 15254098.0,
1590
+ "step": 158
1591
+ },
1592
+ {
1593
+ "entropy": 0.96369419246912,
1594
+ "epoch": 1.5326876513317191,
1595
+ "grad_norm": 0.17047317326068878,
1596
+ "learning_rate": 2.659094751063666e-05,
1597
+ "loss": 0.951930046081543,
1598
+ "mean_token_accuracy": 0.7086149156093597,
1599
+ "num_tokens": 15351681.0,
1600
+ "step": 159
1601
+ },
1602
+ {
1603
+ "entropy": 0.9510526731610298,
1604
+ "epoch": 1.542372881355932,
1605
+ "grad_norm": 0.16374436020851135,
1606
+ "learning_rate": 2.6326063482536934e-05,
1607
+ "loss": 0.9372283816337585,
1608
+ "mean_token_accuracy": 0.7110410034656525,
1609
+ "num_tokens": 15449368.0,
1610
+ "step": 160
1611
+ },
1612
+ {
1613
+ "entropy": 0.9636484161019325,
1614
+ "epoch": 1.5520581113801453,
1615
+ "grad_norm": 0.16882875561714172,
1616
+ "learning_rate": 2.606103007990371e-05,
1617
+ "loss": 0.9486697912216187,
1618
+ "mean_token_accuracy": 0.7095256149768829,
1619
+ "num_tokens": 15546838.0,
1620
+ "step": 161
1621
+ },
1622
+ {
1623
+ "entropy": 0.9014792814850807,
1624
+ "epoch": 1.5617433414043584,
1625
+ "grad_norm": 0.16102053225040436,
1626
+ "learning_rate": 2.579587715744712e-05,
1627
+ "loss": 0.9012171626091003,
1628
+ "mean_token_accuracy": 0.7217540591955185,
1629
+ "num_tokens": 15643978.0,
1630
+ "step": 162
1631
+ },
1632
+ {
1633
+ "entropy": 0.9074290469288826,
1634
+ "epoch": 1.5714285714285714,
1635
+ "grad_norm": 0.16165439784526825,
1636
+ "learning_rate": 2.5530634583340592e-05,
1637
+ "loss": 0.9150861501693726,
1638
+ "mean_token_accuracy": 0.7191483452916145,
1639
+ "num_tokens": 15741798.0,
1640
+ "step": 163
1641
+ },
1642
+ {
1643
+ "entropy": 0.8977201282978058,
1644
+ "epoch": 1.5811138014527844,
1645
+ "grad_norm": 0.16587872803211212,
1646
+ "learning_rate": 2.526533223585641e-05,
1647
+ "loss": 0.8978846073150635,
1648
+ "mean_token_accuracy": 0.7233535423874855,
1649
+ "num_tokens": 15839214.0,
1650
+ "step": 164
1651
+ },
1652
+ {
1653
+ "entropy": 0.9248517453670502,
1654
+ "epoch": 1.5907990314769975,
1655
+ "grad_norm": 0.16850240528583527,
1656
+ "learning_rate": 2.5e-05,
1657
+ "loss": 0.9193021655082703,
1658
+ "mean_token_accuracy": 0.7177055552601814,
1659
+ "num_tokens": 15937070.0,
1660
+ "step": 165
1661
+ },
1662
+ {
1663
+ "entropy": 0.947799064218998,
1664
+ "epoch": 1.6004842615012107,
1665
+ "grad_norm": 0.1697651892900467,
1666
+ "learning_rate": 2.47346677641436e-05,
1667
+ "loss": 0.9431581497192383,
1668
+ "mean_token_accuracy": 0.7087105363607407,
1669
+ "num_tokens": 16034036.0,
1670
+ "step": 166
1671
+ },
1672
+ {
1673
+ "entropy": 0.919052854180336,
1674
+ "epoch": 1.6101694915254239,
1675
+ "grad_norm": 0.16670337319374084,
1676
+ "learning_rate": 2.446936541665941e-05,
1677
+ "loss": 0.909469485282898,
1678
+ "mean_token_accuracy": 0.7194403260946274,
1679
+ "num_tokens": 16130526.0,
1680
+ "step": 167
1681
+ },
1682
+ {
1683
+ "entropy": 0.9474556148052216,
1684
+ "epoch": 1.6198547215496368,
1685
+ "grad_norm": 0.17450463771820068,
1686
+ "learning_rate": 2.4204122842552896e-05,
1687
+ "loss": 0.94859379529953,
1688
+ "mean_token_accuracy": 0.7095327079296112,
1689
+ "num_tokens": 16226871.0,
1690
+ "step": 168
1691
+ },
1692
+ {
1693
+ "entropy": 0.913779616355896,
1694
+ "epoch": 1.6295399515738498,
1695
+ "grad_norm": 0.17406359314918518,
1696
+ "learning_rate": 2.39389699200963e-05,
1697
+ "loss": 0.907940149307251,
1698
+ "mean_token_accuracy": 0.7214842140674591,
1699
+ "num_tokens": 16323264.0,
1700
+ "step": 169
1701
+ },
1702
+ {
1703
+ "entropy": 0.9468366056680679,
1704
+ "epoch": 1.639225181598063,
1705
+ "grad_norm": 0.16962678730487823,
1706
+ "learning_rate": 2.3673936517463075e-05,
1707
+ "loss": 0.9424924850463867,
1708
+ "mean_token_accuracy": 0.7086934819817543,
1709
+ "num_tokens": 16420896.0,
1710
+ "step": 170
1711
+ },
1712
+ {
1713
+ "entropy": 0.9031177535653114,
1714
+ "epoch": 1.6489104116222761,
1715
+ "grad_norm": 0.16391068696975708,
1716
+ "learning_rate": 2.3409052489363342e-05,
1717
+ "loss": 0.8965908885002136,
1718
+ "mean_token_accuracy": 0.7222048938274384,
1719
+ "num_tokens": 16517596.0,
1720
+ "step": 171
1721
+ },
1722
+ {
1723
+ "entropy": 0.9283922016620636,
1724
+ "epoch": 1.658595641646489,
1725
+ "grad_norm": 0.1656733751296997,
1726
+ "learning_rate": 2.3144347673680936e-05,
1727
+ "loss": 0.9119100570678711,
1728
+ "mean_token_accuracy": 0.7196318805217743,
1729
+ "num_tokens": 16614396.0,
1730
+ "step": 172
1731
+ },
1732
+ {
1733
+ "entropy": 0.9282249063253403,
1734
+ "epoch": 1.668280871670702,
1735
+ "grad_norm": 0.1733785718679428,
1736
+ "learning_rate": 2.287985188811228e-05,
1737
+ "loss": 0.909268856048584,
1738
+ "mean_token_accuracy": 0.7201012820005417,
1739
+ "num_tokens": 16711397.0,
1740
+ "step": 173
1741
+ },
1742
+ {
1743
+ "entropy": 0.9379151538014412,
1744
+ "epoch": 1.6779661016949152,
1745
+ "grad_norm": 0.16747304797172546,
1746
+ "learning_rate": 2.261559492680755e-05,
1747
+ "loss": 0.9251323938369751,
1748
+ "mean_token_accuracy": 0.716295875608921,
1749
+ "num_tokens": 16808068.0,
1750
+ "step": 174
1751
+ },
1752
+ {
1753
+ "entropy": 0.9612104222178459,
1754
+ "epoch": 1.6876513317191284,
1755
+ "grad_norm": 0.1707066297531128,
1756
+ "learning_rate": 2.23516065570146e-05,
1757
+ "loss": 0.9485330581665039,
1758
+ "mean_token_accuracy": 0.7085242122411728,
1759
+ "num_tokens": 16904236.0,
1760
+ "step": 175
1761
+ },
1762
+ {
1763
+ "entropy": 0.9023498296737671,
1764
+ "epoch": 1.6973365617433414,
1765
+ "grad_norm": 0.17008283734321594,
1766
+ "learning_rate": 2.2087916515725737e-05,
1767
+ "loss": 0.8945733308792114,
1768
+ "mean_token_accuracy": 0.726103164255619,
1769
+ "num_tokens": 17000607.0,
1770
+ "step": 176
1771
+ },
1772
+ {
1773
+ "entropy": 0.9537338986992836,
1774
+ "epoch": 1.7070217917675545,
1775
+ "grad_norm": 0.16386950016021729,
1776
+ "learning_rate": 2.182455450632803e-05,
1777
+ "loss": 0.9525269865989685,
1778
+ "mean_token_accuracy": 0.7078746929764748,
1779
+ "num_tokens": 17098544.0,
1780
+ "step": 177
1781
+ },
1782
+ {
1783
+ "entropy": 0.9302299544215202,
1784
+ "epoch": 1.7167070217917675,
1785
+ "grad_norm": 0.16354598104953766,
1786
+ "learning_rate": 2.1561550195257378e-05,
1787
+ "loss": 0.9283109903335571,
1788
+ "mean_token_accuracy": 0.7142400592565536,
1789
+ "num_tokens": 17196350.0,
1790
+ "step": 178
1791
+ },
1792
+ {
1793
+ "entropy": 0.9149316698312759,
1794
+ "epoch": 1.7263922518159807,
1795
+ "grad_norm": 0.17065662145614624,
1796
+ "learning_rate": 2.1298933208656718e-05,
1797
+ "loss": 0.926344633102417,
1798
+ "mean_token_accuracy": 0.7150438651442528,
1799
+ "num_tokens": 17293747.0,
1800
+ "step": 179
1801
+ },
1802
+ {
1803
+ "entropy": 0.9035502597689629,
1804
+ "epoch": 1.7360774818401938,
1805
+ "grad_norm": 0.16498662531375885,
1806
+ "learning_rate": 2.103673312903878e-05,
1807
+ "loss": 0.9038597345352173,
1808
+ "mean_token_accuracy": 0.7208788171410561,
1809
+ "num_tokens": 17390350.0,
1810
+ "step": 180
1811
+ },
1812
+ {
1813
+ "entropy": 0.9007836878299713,
1814
+ "epoch": 1.7457627118644068,
1815
+ "grad_norm": 0.15899653732776642,
1816
+ "learning_rate": 2.0774979491953777e-05,
1817
+ "loss": 0.8981961011886597,
1818
+ "mean_token_accuracy": 0.7238594442605972,
1819
+ "num_tokens": 17488569.0,
1820
+ "step": 181
1821
+ },
1822
+ {
1823
+ "entropy": 0.9600996002554893,
1824
+ "epoch": 1.7554479418886197,
1825
+ "grad_norm": 0.16713890433311462,
1826
+ "learning_rate": 2.0513701782662366e-05,
1827
+ "loss": 0.9571723937988281,
1828
+ "mean_token_accuracy": 0.7069702818989754,
1829
+ "num_tokens": 17585738.0,
1830
+ "step": 182
1831
+ },
1832
+ {
1833
+ "entropy": 0.9352350756525993,
1834
+ "epoch": 1.765133171912833,
1835
+ "grad_norm": 0.16733741760253906,
1836
+ "learning_rate": 2.025292943281429e-05,
1837
+ "loss": 0.9361134767532349,
1838
+ "mean_token_accuracy": 0.7141696363687515,
1839
+ "num_tokens": 17679675.0,
1840
+ "step": 183
1841
+ },
1842
+ {
1843
+ "entropy": 0.9395305886864662,
1844
+ "epoch": 1.774818401937046,
1845
+ "grad_norm": 0.17384639382362366,
1846
+ "learning_rate": 1.9992691817133024e-05,
1847
+ "loss": 0.9297868609428406,
1848
+ "mean_token_accuracy": 0.7141037434339523,
1849
+ "num_tokens": 17776988.0,
1850
+ "step": 184
1851
+ },
1852
+ {
1853
+ "entropy": 0.9624091535806656,
1854
+ "epoch": 1.784503631961259,
1855
+ "grad_norm": 0.16501930356025696,
1856
+ "learning_rate": 1.973301825010685e-05,
1857
+ "loss": 0.9562557935714722,
1858
+ "mean_token_accuracy": 0.7060800492763519,
1859
+ "num_tokens": 17874853.0,
1860
+ "step": 185
1861
+ },
1862
+ {
1863
+ "entropy": 0.9264740124344826,
1864
+ "epoch": 1.794188861985472,
1865
+ "grad_norm": 0.19295631349086761,
1866
+ "learning_rate": 1.947393798268676e-05,
1867
+ "loss": 0.9122995138168335,
1868
+ "mean_token_accuracy": 0.7201044335961342,
1869
+ "num_tokens": 17972657.0,
1870
+ "step": 186
1871
+ },
1872
+ {
1873
+ "entropy": 0.947826087474823,
1874
+ "epoch": 1.8038740920096852,
1875
+ "grad_norm": 0.16467559337615967,
1876
+ "learning_rate": 1.9215480198991466e-05,
1877
+ "loss": 0.9421426057815552,
1878
+ "mean_token_accuracy": 0.7108215615153313,
1879
+ "num_tokens": 18070004.0,
1880
+ "step": 187
1881
+ },
1882
+ {
1883
+ "entropy": 0.9447982832789421,
1884
+ "epoch": 1.8135593220338984,
1885
+ "grad_norm": 0.16559769213199615,
1886
+ "learning_rate": 1.895767401301994e-05,
1887
+ "loss": 0.9430465698242188,
1888
+ "mean_token_accuracy": 0.7096053883433342,
1889
+ "num_tokens": 18167615.0,
1890
+ "step": 188
1891
+ },
1892
+ {
1893
+ "entropy": 0.9548698142170906,
1894
+ "epoch": 1.8232445520581115,
1895
+ "grad_norm": 0.17204411327838898,
1896
+ "learning_rate": 1.8700548465371874e-05,
1897
+ "loss": 0.9438506364822388,
1898
+ "mean_token_accuracy": 0.7104759141802788,
1899
+ "num_tokens": 18264186.0,
1900
+ "step": 189
1901
+ },
1902
+ {
1903
+ "entropy": 0.9263730049133301,
1904
+ "epoch": 1.8329297820823245,
1905
+ "grad_norm": 0.2153022289276123,
1906
+ "learning_rate": 1.84441325199764e-05,
1907
+ "loss": 0.9292491674423218,
1908
+ "mean_token_accuracy": 0.7148165255784988,
1909
+ "num_tokens": 18360549.0,
1910
+ "step": 190
1911
+ },
1912
+ {
1913
+ "entropy": 0.9317336082458496,
1914
+ "epoch": 1.8426150121065374,
1915
+ "grad_norm": 0.16318465769290924,
1916
+ "learning_rate": 1.818845506082943e-05,
1917
+ "loss": 0.9184021353721619,
1918
+ "mean_token_accuracy": 0.7164310812950134,
1919
+ "num_tokens": 18458305.0,
1920
+ "step": 191
1921
+ },
1922
+ {
1923
+ "entropy": 0.953622855246067,
1924
+ "epoch": 1.8523002421307506,
1925
+ "grad_norm": 0.16858021914958954,
1926
+ "learning_rate": 1.793354488874006e-05,
1927
+ "loss": 0.9495086669921875,
1928
+ "mean_token_accuracy": 0.7083204239606857,
1929
+ "num_tokens": 18555499.0,
1930
+ "step": 192
1931
+ },
1932
+ {
1933
+ "entropy": 0.9400290995836258,
1934
+ "epoch": 1.8619854721549638,
1935
+ "grad_norm": 0.17251019179821014,
1936
+ "learning_rate": 1.7679430718086243e-05,
1937
+ "loss": 0.9359773397445679,
1938
+ "mean_token_accuracy": 0.712072916328907,
1939
+ "num_tokens": 18653173.0,
1940
+ "step": 193
1941
+ },
1942
+ {
1943
+ "entropy": 0.9087251797318459,
1944
+ "epoch": 1.8716707021791767,
1945
+ "grad_norm": 0.16202928125858307,
1946
+ "learning_rate": 1.742614117358029e-05,
1947
+ "loss": 0.9024781584739685,
1948
+ "mean_token_accuracy": 0.7245003953576088,
1949
+ "num_tokens": 18749049.0,
1950
+ "step": 194
1951
+ },
1952
+ {
1953
+ "entropy": 0.8747521638870239,
1954
+ "epoch": 1.8813559322033897,
1955
+ "grad_norm": 0.16475841403007507,
1956
+ "learning_rate": 1.7173704787044446e-05,
1957
+ "loss": 0.8768506050109863,
1958
+ "mean_token_accuracy": 0.729314886033535,
1959
+ "num_tokens": 18846779.0,
1960
+ "step": 195
1961
+ },
1962
+ {
1963
+ "entropy": 0.9364713057875633,
1964
+ "epoch": 1.8910411622276029,
1965
+ "grad_norm": 0.17578846216201782,
1966
+ "learning_rate": 1.6922149994196878e-05,
1967
+ "loss": 0.9387744665145874,
1968
+ "mean_token_accuracy": 0.7123603448271751,
1969
+ "num_tokens": 18942612.0,
1970
+ "step": 196
1971
+ },
1972
+ {
1973
+ "entropy": 0.9117182791233063,
1974
+ "epoch": 1.900726392251816,
1975
+ "grad_norm": 0.162763312458992,
1976
+ "learning_rate": 1.667150513144856e-05,
1977
+ "loss": 0.9101252555847168,
1978
+ "mean_token_accuracy": 0.7196433991193771,
1979
+ "num_tokens": 19040612.0,
1980
+ "step": 197
1981
+ },
1982
+ {
1983
+ "entropy": 0.9393728226423264,
1984
+ "epoch": 1.910411622276029,
1985
+ "grad_norm": 0.1870926320552826,
1986
+ "learning_rate": 1.6421798432711345e-05,
1987
+ "loss": 0.9326294660568237,
1988
+ "mean_token_accuracy": 0.7153022885322571,
1989
+ "num_tokens": 19137532.0,
1990
+ "step": 198
1991
+ },
1992
+ {
1993
+ "entropy": 0.9431560337543488,
1994
+ "epoch": 1.9200968523002422,
1995
+ "grad_norm": 0.16632631421089172,
1996
+ "learning_rate": 1.617305802621748e-05,
1997
+ "loss": 0.9342424273490906,
1998
+ "mean_token_accuracy": 0.7112816274166107,
1999
+ "num_tokens": 19235643.0,
2000
+ "step": 199
2001
+ },
2002
+ {
2003
+ "entropy": 0.9072659760713577,
2004
+ "epoch": 1.9297820823244551,
2005
+ "grad_norm": 0.16011442244052887,
2006
+ "learning_rate": 1.5925311931351172e-05,
2007
+ "loss": 0.8909319639205933,
2008
+ "mean_token_accuracy": 0.7232344001531601,
2009
+ "num_tokens": 19333124.0,
2010
+ "step": 200
2011
+ },
2012
+ {
2013
+ "entropy": 0.9128969237208366,
2014
+ "epoch": 1.9394673123486683,
2015
+ "grad_norm": 0.1665753275156021,
2016
+ "learning_rate": 1.567858805549229e-05,
2017
+ "loss": 0.8946795463562012,
2018
+ "mean_token_accuracy": 0.7247551828622818,
2019
+ "num_tokens": 19429815.0,
2020
+ "step": 201
2021
+ },
2022
+ {
2023
+ "entropy": 0.931357629597187,
2024
+ "epoch": 1.9491525423728815,
2025
+ "grad_norm": 0.16533996164798737,
2026
+ "learning_rate": 1.5432914190872757e-05,
2027
+ "loss": 0.921809196472168,
2028
+ "mean_token_accuracy": 0.7162308841943741,
2029
+ "num_tokens": 19526543.0,
2030
+ "step": 202
2031
+ },
2032
+ {
2033
+ "entropy": 0.9272895827889442,
2034
+ "epoch": 1.9588377723970944,
2035
+ "grad_norm": 0.1601615697145462,
2036
+ "learning_rate": 1.5188318011445906e-05,
2037
+ "loss": 0.9188361167907715,
2038
+ "mean_token_accuracy": 0.7175915241241455,
2039
+ "num_tokens": 19624300.0,
2040
+ "step": 203
2041
+ },
2042
+ {
2043
+ "entropy": 0.9179273992776871,
2044
+ "epoch": 1.9685230024213074,
2045
+ "grad_norm": 0.15775485336780548,
2046
+ "learning_rate": 1.4944827069769123e-05,
2047
+ "loss": 0.9079532623291016,
2048
+ "mean_token_accuracy": 0.7213004529476166,
2049
+ "num_tokens": 19721985.0,
2050
+ "step": 204
2051
+ },
2052
+ {
2053
+ "entropy": 0.9033334702253342,
2054
+ "epoch": 1.9782082324455206,
2055
+ "grad_norm": 0.16341419517993927,
2056
+ "learning_rate": 1.4702468793900188e-05,
2057
+ "loss": 0.8908724188804626,
2058
+ "mean_token_accuracy": 0.7247681170701981,
2059
+ "num_tokens": 19818914.0,
2060
+ "step": 205
2061
+ },
2062
+ {
2063
+ "entropy": 0.8941410183906555,
2064
+ "epoch": 1.9878934624697338,
2065
+ "grad_norm": 0.1583692729473114,
2066
+ "learning_rate": 1.4461270484307643e-05,
2067
+ "loss": 0.892493486404419,
2068
+ "mean_token_accuracy": 0.7225205078721046,
2069
+ "num_tokens": 19916341.0,
2070
+ "step": 206
2071
+ },
2072
+ {
2073
+ "entropy": 0.8992219567298889,
2074
+ "epoch": 1.9975786924939467,
2075
+ "grad_norm": 0.1586410403251648,
2076
+ "learning_rate": 1.4221259310795543e-05,
2077
+ "loss": 0.8945289850234985,
2078
+ "mean_token_accuracy": 0.7262753918766975,
2079
+ "num_tokens": 20013246.0,
2080
+ "step": 207
2081
+ },
2082
+ {
2083
+ "entropy": 0.9463706612586975,
2084
+ "epoch": 2.0,
2085
+ "grad_norm": 0.33229002356529236,
2086
+ "learning_rate": 1.3982462309442889e-05,
2087
+ "loss": 0.9369756579399109,
2088
+ "mean_token_accuracy": 0.7111198902130127,
2089
+ "num_tokens": 20037358.0,
2090
+ "step": 208
2091
+ },
2092
+ {
2093
+ "entropy": 0.8963305726647377,
2094
+ "epoch": 2.009685230024213,
2095
+ "grad_norm": 0.16715960204601288,
2096
+ "learning_rate": 1.3744906379558165e-05,
2097
+ "loss": 0.8716245889663696,
2098
+ "mean_token_accuracy": 0.7283544465899467,
2099
+ "num_tokens": 20134872.0,
2100
+ "step": 209
2101
+ },
2102
+ {
2103
+ "entropy": 0.9343423992395401,
2104
+ "epoch": 2.0193704600484264,
2105
+ "grad_norm": 0.16829326748847961,
2106
+ "learning_rate": 1.3508618280649255e-05,
2107
+ "loss": 0.9013060331344604,
2108
+ "mean_token_accuracy": 0.7196413949131966,
2109
+ "num_tokens": 20231994.0,
2110
+ "step": 210
2111
+ },
2112
+ {
2113
+ "entropy": 0.9421586096286774,
2114
+ "epoch": 2.029055690072639,
2115
+ "grad_norm": 0.16777563095092773,
2116
+ "learning_rate": 1.3273624629409082e-05,
2117
+ "loss": 0.9127623438835144,
2118
+ "mean_token_accuracy": 0.7166152596473694,
2119
+ "num_tokens": 20329201.0,
2120
+ "step": 211
2121
+ },
2122
+ {
2123
+ "entropy": 0.8845984265208244,
2124
+ "epoch": 2.0387409200968523,
2125
+ "grad_norm": 0.1653762012720108,
2126
+ "learning_rate": 1.3039951896717445e-05,
2127
+ "loss": 0.8587235808372498,
2128
+ "mean_token_accuracy": 0.7311449274420738,
2129
+ "num_tokens": 20425962.0,
2130
+ "step": 212
2131
+ },
2132
+ {
2133
+ "entropy": 0.9084889963269234,
2134
+ "epoch": 2.0484261501210654,
2135
+ "grad_norm": 0.17060290277004242,
2136
+ "learning_rate": 1.2807626404659142e-05,
2137
+ "loss": 0.8947319984436035,
2138
+ "mean_token_accuracy": 0.7232846319675446,
2139
+ "num_tokens": 20521440.0,
2140
+ "step": 213
2141
+ },
2142
+ {
2143
+ "entropy": 0.8783224523067474,
2144
+ "epoch": 2.0581113801452786,
2145
+ "grad_norm": 0.1699601113796234,
2146
+ "learning_rate": 1.2576674323558929e-05,
2147
+ "loss": 0.8778979182243347,
2148
+ "mean_token_accuracy": 0.7268319725990295,
2149
+ "num_tokens": 20619455.0,
2150
+ "step": 214
2151
+ },
2152
+ {
2153
+ "entropy": 0.8795689567923546,
2154
+ "epoch": 2.0677966101694913,
2155
+ "grad_norm": 0.17277944087982178,
2156
+ "learning_rate": 1.234712166903359e-05,
2157
+ "loss": 0.8746508359909058,
2158
+ "mean_token_accuracy": 0.7274192124605179,
2159
+ "num_tokens": 20715972.0,
2160
+ "step": 215
2161
+ },
2162
+ {
2163
+ "entropy": 0.8955965638160706,
2164
+ "epoch": 2.0774818401937045,
2165
+ "grad_norm": 0.16799940168857574,
2166
+ "learning_rate": 1.2118994299061376e-05,
2167
+ "loss": 0.8971080780029297,
2168
+ "mean_token_accuracy": 0.7206688448786736,
2169
+ "num_tokens": 20813191.0,
2170
+ "step": 216
2171
+ },
2172
+ {
2173
+ "entropy": 0.8811242133378983,
2174
+ "epoch": 2.0871670702179177,
2175
+ "grad_norm": 0.17031392455101013,
2176
+ "learning_rate": 1.1892317911069212e-05,
2177
+ "loss": 0.8863786458969116,
2178
+ "mean_token_accuracy": 0.7245300561189651,
2179
+ "num_tokens": 20910966.0,
2180
+ "step": 217
2181
+ },
2182
+ {
2183
+ "entropy": 0.8759906217455864,
2184
+ "epoch": 2.096852300242131,
2185
+ "grad_norm": 0.174867182970047,
2186
+ "learning_rate": 1.1667118039038064e-05,
2187
+ "loss": 0.8835239410400391,
2188
+ "mean_token_accuracy": 0.7252249047160149,
2189
+ "num_tokens": 21008138.0,
2190
+ "step": 218
2191
+ },
2192
+ {
2193
+ "entropy": 0.8668298348784447,
2194
+ "epoch": 2.106537530266344,
2195
+ "grad_norm": 0.17496958374977112,
2196
+ "learning_rate": 1.1443420050626625e-05,
2197
+ "loss": 0.8708558082580566,
2198
+ "mean_token_accuracy": 0.728722371160984,
2199
+ "num_tokens": 21104696.0,
2200
+ "step": 219
2201
+ },
2202
+ {
2203
+ "entropy": 0.9141282364726067,
2204
+ "epoch": 2.116222760290557,
2205
+ "grad_norm": 0.18080434203147888,
2206
+ "learning_rate": 1.1221249144313778e-05,
2207
+ "loss": 0.922538161277771,
2208
+ "mean_token_accuracy": 0.7147187516093254,
2209
+ "num_tokens": 21201964.0,
2210
+ "step": 220
2211
+ },
2212
+ {
2213
+ "entropy": 0.8473684266209602,
2214
+ "epoch": 2.12590799031477,
2215
+ "grad_norm": 0.1684316247701645,
2216
+ "learning_rate": 1.1000630346560117e-05,
2217
+ "loss": 0.845892608165741,
2218
+ "mean_token_accuracy": 0.738263763487339,
2219
+ "num_tokens": 21297694.0,
2220
+ "step": 221
2221
+ },
2222
+ {
2223
+ "entropy": 0.8640781044960022,
2224
+ "epoch": 2.135593220338983,
2225
+ "grad_norm": 0.17115196585655212,
2226
+ "learning_rate": 1.078158850898883e-05,
2227
+ "loss": 0.8640755414962769,
2228
+ "mean_token_accuracy": 0.730094663798809,
2229
+ "num_tokens": 21393691.0,
2230
+ "step": 222
2231
+ },
2232
+ {
2233
+ "entropy": 0.8822661563754082,
2234
+ "epoch": 2.1452784503631963,
2235
+ "grad_norm": 0.1670716106891632,
2236
+ "learning_rate": 1.0564148305586296e-05,
2237
+ "loss": 0.8770127296447754,
2238
+ "mean_token_accuracy": 0.7259002774953842,
2239
+ "num_tokens": 21490650.0,
2240
+ "step": 223
2241
+ },
2242
+ {
2243
+ "entropy": 0.90352463722229,
2244
+ "epoch": 2.154963680387409,
2245
+ "grad_norm": 0.16406244039535522,
2246
+ "learning_rate": 1.0348334229922677e-05,
2247
+ "loss": 0.889420211315155,
2248
+ "mean_token_accuracy": 0.7213627323508263,
2249
+ "num_tokens": 21587982.0,
2250
+ "step": 224
2251
+ },
2252
+ {
2253
+ "entropy": 0.8884696215391159,
2254
+ "epoch": 2.164648910411622,
2255
+ "grad_norm": 0.17088951170444489,
2256
+ "learning_rate": 1.0134170592392836e-05,
2257
+ "loss": 0.8753219842910767,
2258
+ "mean_token_accuracy": 0.7271528840065002,
2259
+ "num_tokens": 21681367.0,
2260
+ "step": 225
2261
+ },
2262
+ {
2263
+ "entropy": 0.9012778177857399,
2264
+ "epoch": 2.1743341404358354,
2265
+ "grad_norm": 0.1680404394865036,
2266
+ "learning_rate": 9.9216815174779e-06,
2267
+ "loss": 0.8923549652099609,
2268
+ "mean_token_accuracy": 0.7228792309761047,
2269
+ "num_tokens": 21778139.0,
2270
+ "step": 226
2271
+ },
2272
+ {
2273
+ "entropy": 0.9223925024271011,
2274
+ "epoch": 2.1840193704600486,
2275
+ "grad_norm": 0.1729201227426529,
2276
+ "learning_rate": 9.710890941027722e-06,
2277
+ "loss": 0.8943996429443359,
2278
+ "mean_token_accuracy": 0.7211974635720253,
2279
+ "num_tokens": 21875544.0,
2280
+ "step": 227
2281
+ },
2282
+ {
2283
+ "entropy": 0.8739941567182541,
2284
+ "epoch": 2.1937046004842613,
2285
+ "grad_norm": 0.17046649754047394,
2286
+ "learning_rate": 9.501822607564678e-06,
2287
+ "loss": 0.8558517694473267,
2288
+ "mean_token_accuracy": 0.7340629026293755,
2289
+ "num_tokens": 21972613.0,
2290
+ "step": 228
2291
+ },
2292
+ {
2293
+ "entropy": 0.8621426746249199,
2294
+ "epoch": 2.2033898305084745,
2295
+ "grad_norm": 0.16286900639533997,
2296
+ "learning_rate": 9.29450006760894e-06,
2297
+ "loss": 0.8458216190338135,
2298
+ "mean_token_accuracy": 0.7358392179012299,
2299
+ "num_tokens": 22069904.0,
2300
+ "step": 229
2301
+ },
2302
+ {
2303
+ "entropy": 0.8863791599869728,
2304
+ "epoch": 2.2130750605326877,
2305
+ "grad_norm": 0.16779263317584991,
2306
+ "learning_rate": 9.088946675025605e-06,
2307
+ "loss": 0.869725227355957,
2308
+ "mean_token_accuracy": 0.7280366569757462,
2309
+ "num_tokens": 22166841.0,
2310
+ "step": 230
2311
+ },
2312
+ {
2313
+ "entropy": 0.8678382188081741,
2314
+ "epoch": 2.222760290556901,
2315
+ "grad_norm": 0.1676686406135559,
2316
+ "learning_rate": 8.88518558439402e-06,
2317
+ "loss": 0.8613146543502808,
2318
+ "mean_token_accuracy": 0.7329499870538712,
2319
+ "num_tokens": 22264126.0,
2320
+ "step": 231
2321
+ },
2322
+ {
2323
+ "entropy": 0.931030385196209,
2324
+ "epoch": 2.232445520581114,
2325
+ "grad_norm": 0.17840994894504547,
2326
+ "learning_rate": 8.683239748399539e-06,
2327
+ "loss": 0.9252626299858093,
2328
+ "mean_token_accuracy": 0.7149468436837196,
2329
+ "num_tokens": 22360793.0,
2330
+ "step": 232
2331
+ },
2332
+ {
2333
+ "entropy": 0.8712918534874916,
2334
+ "epoch": 2.2421307506053267,
2335
+ "grad_norm": 0.16877689957618713,
2336
+ "learning_rate": 8.483131915247968e-06,
2337
+ "loss": 0.8701798915863037,
2338
+ "mean_token_accuracy": 0.7275582626461983,
2339
+ "num_tokens": 22457944.0,
2340
+ "step": 233
2341
+ },
2342
+ {
2343
+ "entropy": 0.8893302232027054,
2344
+ "epoch": 2.25181598062954,
2345
+ "grad_norm": 0.17221516370773315,
2346
+ "learning_rate": 8.284884626103165e-06,
2347
+ "loss": 0.8809248805046082,
2348
+ "mean_token_accuracy": 0.725702092051506,
2349
+ "num_tokens": 22555415.0,
2350
+ "step": 234
2351
+ },
2352
+ {
2353
+ "entropy": 0.9248282313346863,
2354
+ "epoch": 2.261501210653753,
2355
+ "grad_norm": 0.16808673739433289,
2356
+ "learning_rate": 8.088520212547831e-06,
2357
+ "loss": 0.9174758791923523,
2358
+ "mean_token_accuracy": 0.7162574678659439,
2359
+ "num_tokens": 22651864.0,
2360
+ "step": 235
2361
+ },
2362
+ {
2363
+ "entropy": 0.9000514894723892,
2364
+ "epoch": 2.2711864406779663,
2365
+ "grad_norm": 0.1721772998571396,
2366
+ "learning_rate": 7.894060794067987e-06,
2367
+ "loss": 0.8979053497314453,
2368
+ "mean_token_accuracy": 0.7211757078766823,
2369
+ "num_tokens": 22748451.0,
2370
+ "step": 236
2371
+ },
2372
+ {
2373
+ "entropy": 0.9153964892029762,
2374
+ "epoch": 2.280871670702179,
2375
+ "grad_norm": 0.2046622633934021,
2376
+ "learning_rate": 7.701528275561348e-06,
2377
+ "loss": 0.9202542304992676,
2378
+ "mean_token_accuracy": 0.7141179367899895,
2379
+ "num_tokens": 22845789.0,
2380
+ "step": 237
2381
+ },
2382
+ {
2383
+ "entropy": 0.8642693385481834,
2384
+ "epoch": 2.290556900726392,
2385
+ "grad_norm": 0.16806447505950928,
2386
+ "learning_rate": 7.510944344869774e-06,
2387
+ "loss": 0.8547899127006531,
2388
+ "mean_token_accuracy": 0.735467717051506,
2389
+ "num_tokens": 22942730.0,
2390
+ "step": 238
2391
+ },
2392
+ {
2393
+ "entropy": 0.8776782006025314,
2394
+ "epoch": 2.3002421307506054,
2395
+ "grad_norm": 0.1855197250843048,
2396
+ "learning_rate": 7.3223304703363135e-06,
2397
+ "loss": 0.8706147074699402,
2398
+ "mean_token_accuracy": 0.7279384657740593,
2399
+ "num_tokens": 23040618.0,
2400
+ "step": 239
2401
+ },
2402
+ {
2403
+ "entropy": 0.8683270290493965,
2404
+ "epoch": 2.3099273607748185,
2405
+ "grad_norm": 0.16676853597164154,
2406
+ "learning_rate": 7.13570789838687e-06,
2407
+ "loss": 0.8610081076622009,
2408
+ "mean_token_accuracy": 0.7308985963463783,
2409
+ "num_tokens": 23138065.0,
2410
+ "step": 240
2411
+ },
2412
+ {
2413
+ "entropy": 0.8782318234443665,
2414
+ "epoch": 2.3196125907990313,
2415
+ "grad_norm": 0.1693039983510971,
2416
+ "learning_rate": 6.951097651136889e-06,
2417
+ "loss": 0.8674857020378113,
2418
+ "mean_token_accuracy": 0.728962704539299,
2419
+ "num_tokens": 23235205.0,
2420
+ "step": 241
2421
+ },
2422
+ {
2423
+ "entropy": 0.8897048607468605,
2424
+ "epoch": 2.3292978208232444,
2425
+ "grad_norm": 0.17293794453144073,
2426
+ "learning_rate": 6.768520524023348e-06,
2427
+ "loss": 0.8793017268180847,
2428
+ "mean_token_accuracy": 0.7251008674502373,
2429
+ "num_tokens": 23331969.0,
2430
+ "step": 242
2431
+ },
2432
+ {
2433
+ "entropy": 0.8846061527729034,
2434
+ "epoch": 2.3389830508474576,
2435
+ "grad_norm": 0.16847588121891022,
2436
+ "learning_rate": 6.587997083462197e-06,
2437
+ "loss": 0.8802613019943237,
2438
+ "mean_token_accuracy": 0.7245057225227356,
2439
+ "num_tokens": 23429634.0,
2440
+ "step": 243
2441
+ },
2442
+ {
2443
+ "entropy": 0.9211963415145874,
2444
+ "epoch": 2.348668280871671,
2445
+ "grad_norm": 0.20498435199260712,
2446
+ "learning_rate": 6.4095476645317346e-06,
2447
+ "loss": 0.9113345146179199,
2448
+ "mean_token_accuracy": 0.7170490995049477,
2449
+ "num_tokens": 23527286.0,
2450
+ "step": 244
2451
+ },
2452
+ {
2453
+ "entropy": 0.8628609254956245,
2454
+ "epoch": 2.358353510895884,
2455
+ "grad_norm": 0.16568608582019806,
2456
+ "learning_rate": 6.23319236868189e-06,
2457
+ "loss": 0.8587435483932495,
2458
+ "mean_token_accuracy": 0.7304293736815453,
2459
+ "num_tokens": 23624559.0,
2460
+ "step": 245
2461
+ },
2462
+ {
2463
+ "entropy": 0.8532272204756737,
2464
+ "epoch": 2.3680387409200967,
2465
+ "grad_norm": 0.1692337989807129,
2466
+ "learning_rate": 6.058951061469931e-06,
2467
+ "loss": 0.850928783416748,
2468
+ "mean_token_accuracy": 0.7356282025575638,
2469
+ "num_tokens": 23720888.0,
2470
+ "step": 246
2471
+ },
2472
+ {
2473
+ "entropy": 0.8992898389697075,
2474
+ "epoch": 2.37772397094431,
2475
+ "grad_norm": 0.170456662774086,
2476
+ "learning_rate": 5.886843370322692e-06,
2477
+ "loss": 0.8954350352287292,
2478
+ "mean_token_accuracy": 0.7213400527834892,
2479
+ "num_tokens": 23817332.0,
2480
+ "step": 247
2481
+ },
2482
+ {
2483
+ "entropy": 0.8801911473274231,
2484
+ "epoch": 2.387409200968523,
2485
+ "grad_norm": 0.8809273838996887,
2486
+ "learning_rate": 5.7168886823256355e-06,
2487
+ "loss": 0.8754285573959351,
2488
+ "mean_token_accuracy": 0.7273881211876869,
2489
+ "num_tokens": 23914942.0,
2490
+ "step": 248
2491
+ },
2492
+ {
2493
+ "entropy": 0.8864389285445213,
2494
+ "epoch": 2.3970944309927362,
2495
+ "grad_norm": 0.16699263453483582,
2496
+ "learning_rate": 5.549106142039018e-06,
2497
+ "loss": 0.883417010307312,
2498
+ "mean_token_accuracy": 0.7244465425610542,
2499
+ "num_tokens": 24012235.0,
2500
+ "step": 249
2501
+ },
2502
+ {
2503
+ "entropy": 0.8792407065629959,
2504
+ "epoch": 2.406779661016949,
2505
+ "grad_norm": 0.1641698032617569,
2506
+ "learning_rate": 5.38351464934132e-06,
2507
+ "loss": 0.8722311854362488,
2508
+ "mean_token_accuracy": 0.7289138287305832,
2509
+ "num_tokens": 24109746.0,
2510
+ "step": 250
2511
+ },
2512
+ {
2513
+ "entropy": 0.8531442359089851,
2514
+ "epoch": 2.416464891041162,
2515
+ "grad_norm": 0.16733677685260773,
2516
+ "learning_rate": 5.220132857300286e-06,
2517
+ "loss": 0.844599723815918,
2518
+ "mean_token_accuracy": 0.7346237674355507,
2519
+ "num_tokens": 24206911.0,
2520
+ "step": 251
2521
+ },
2522
+ {
2523
+ "entropy": 0.8622922822833061,
2524
+ "epoch": 2.4261501210653753,
2525
+ "grad_norm": 0.16506442427635193,
2526
+ "learning_rate": 5.058979170071754e-06,
2527
+ "loss": 0.8556405901908875,
2528
+ "mean_token_accuracy": 0.7325782477855682,
2529
+ "num_tokens": 24303050.0,
2530
+ "step": 252
2531
+ },
2532
+ {
2533
+ "entropy": 0.8732071667909622,
2534
+ "epoch": 2.4358353510895885,
2535
+ "grad_norm": 0.16514050960540771,
2536
+ "learning_rate": 4.900071740826489e-06,
2537
+ "loss": 0.8675801157951355,
2538
+ "mean_token_accuracy": 0.7301812469959259,
2539
+ "num_tokens": 24400267.0,
2540
+ "step": 253
2541
+ },
2542
+ {
2543
+ "entropy": 0.8459499776363373,
2544
+ "epoch": 2.4455205811138017,
2545
+ "grad_norm": 0.1664392054080963,
2546
+ "learning_rate": 4.743428469705335e-06,
2547
+ "loss": 0.8367156982421875,
2548
+ "mean_token_accuracy": 0.7390483319759369,
2549
+ "num_tokens": 24497596.0,
2550
+ "step": 254
2551
+ },
2552
+ {
2553
+ "entropy": 0.8935896828770638,
2554
+ "epoch": 2.4552058111380144,
2555
+ "grad_norm": 0.1694948524236679,
2556
+ "learning_rate": 4.58906700180286e-06,
2557
+ "loss": 0.8806345462799072,
2558
+ "mean_token_accuracy": 0.7257685586810112,
2559
+ "num_tokens": 24595150.0,
2560
+ "step": 255
2561
+ },
2562
+ {
2563
+ "entropy": 0.8856732323765755,
2564
+ "epoch": 2.4648910411622276,
2565
+ "grad_norm": 0.16689422726631165,
2566
+ "learning_rate": 4.437004725179713e-06,
2567
+ "loss": 0.8796195387840271,
2568
+ "mean_token_accuracy": 0.7265753746032715,
2569
+ "num_tokens": 24692011.0,
2570
+ "step": 256
2571
+ },
2572
+ {
2573
+ "entropy": 0.8778555989265442,
2574
+ "epoch": 2.4745762711864407,
2575
+ "grad_norm": 0.16599559783935547,
2576
+ "learning_rate": 4.2872587689039484e-06,
2577
+ "loss": 0.8729972839355469,
2578
+ "mean_token_accuracy": 0.7282434031367302,
2579
+ "num_tokens": 24789179.0,
2580
+ "step": 257
2581
+ },
2582
+ {
2583
+ "entropy": 0.8913936242461205,
2584
+ "epoch": 2.484261501210654,
2585
+ "grad_norm": 0.16762840747833252,
2586
+ "learning_rate": 4.139846001121514e-06,
2587
+ "loss": 0.8818777799606323,
2588
+ "mean_token_accuracy": 0.7246523946523666,
2589
+ "num_tokens": 24886814.0,
2590
+ "step": 258
2591
+ },
2592
+ {
2593
+ "entropy": 0.8659812286496162,
2594
+ "epoch": 2.4939467312348667,
2595
+ "grad_norm": 0.16921506822109222,
2596
+ "learning_rate": 3.9947830271561435e-06,
2597
+ "loss": 0.8583054542541504,
2598
+ "mean_token_accuracy": 0.7309543639421463,
2599
+ "num_tokens": 24984172.0,
2600
+ "step": 259
2601
+ },
2602
+ {
2603
+ "entropy": 0.877836674451828,
2604
+ "epoch": 2.50363196125908,
2605
+ "grad_norm": 0.16596776247024536,
2606
+ "learning_rate": 3.852086187638846e-06,
2607
+ "loss": 0.8676304221153259,
2608
+ "mean_token_accuracy": 0.727939061820507,
2609
+ "num_tokens": 25082047.0,
2610
+ "step": 260
2611
+ },
2612
+ {
2613
+ "entropy": 0.8626164346933365,
2614
+ "epoch": 2.513317191283293,
2615
+ "grad_norm": 0.16234047710895538,
2616
+ "learning_rate": 3.711771556667218e-06,
2617
+ "loss": 0.8534390330314636,
2618
+ "mean_token_accuracy": 0.7333679273724556,
2619
+ "num_tokens": 25179554.0,
2620
+ "step": 261
2621
+ },
2622
+ {
2623
+ "entropy": 0.8966336399316788,
2624
+ "epoch": 2.523002421307506,
2625
+ "grad_norm": 0.16956865787506104,
2626
+ "learning_rate": 3.57385493999477e-06,
2627
+ "loss": 0.8891651034355164,
2628
+ "mean_token_accuracy": 0.7216921523213387,
2629
+ "num_tokens": 25276660.0,
2630
+ "step": 262
2631
+ },
2632
+ {
2633
+ "entropy": 0.8567296415567398,
2634
+ "epoch": 2.5326876513317194,
2635
+ "grad_norm": 0.16595657169818878,
2636
+ "learning_rate": 3.438351873250492e-06,
2637
+ "loss": 0.8461240530014038,
2638
+ "mean_token_accuracy": 0.735106885433197,
2639
+ "num_tokens": 25374065.0,
2640
+ "step": 263
2641
+ },
2642
+ {
2643
+ "entropy": 0.8628156557679176,
2644
+ "epoch": 2.542372881355932,
2645
+ "grad_norm": 0.17150571942329407,
2646
+ "learning_rate": 3.305277620188826e-06,
2647
+ "loss": 0.8651909232139587,
2648
+ "mean_token_accuracy": 0.7301968634128571,
2649
+ "num_tokens": 25471365.0,
2650
+ "step": 264
2651
+ },
2652
+ {
2653
+ "entropy": 0.9038972184062004,
2654
+ "epoch": 2.5520581113801453,
2655
+ "grad_norm": 0.18031489849090576,
2656
+ "learning_rate": 3.1746471709702964e-06,
2657
+ "loss": 0.8980115652084351,
2658
+ "mean_token_accuracy": 0.7199920937418938,
2659
+ "num_tokens": 25568616.0,
2660
+ "step": 265
2661
+ },
2662
+ {
2663
+ "entropy": 0.8724255040287971,
2664
+ "epoch": 2.5617433414043584,
2665
+ "grad_norm": 0.16530050337314606,
2666
+ "learning_rate": 3.0464752404729484e-06,
2667
+ "loss": 0.8651027679443359,
2668
+ "mean_token_accuracy": 0.7298253774642944,
2669
+ "num_tokens": 25666244.0,
2670
+ "step": 266
2671
+ },
2672
+ {
2673
+ "entropy": 0.8759204596281052,
2674
+ "epoch": 2.571428571428571,
2675
+ "grad_norm": 0.1723172813653946,
2676
+ "learning_rate": 2.920776266634767e-06,
2677
+ "loss": 0.869670569896698,
2678
+ "mean_token_accuracy": 0.7283225506544113,
2679
+ "num_tokens": 25763736.0,
2680
+ "step": 267
2681
+ },
2682
+ {
2683
+ "entropy": 0.8707826510071754,
2684
+ "epoch": 2.5811138014527844,
2685
+ "grad_norm": 0.16445328295230865,
2686
+ "learning_rate": 2.797564408827319e-06,
2687
+ "loss": 0.8592137098312378,
2688
+ "mean_token_accuracy": 0.7309490218758583,
2689
+ "num_tokens": 25860969.0,
2690
+ "step": 268
2691
+ },
2692
+ {
2693
+ "entropy": 0.8872850239276886,
2694
+ "epoch": 2.5907990314769975,
2695
+ "grad_norm": 0.16793842613697052,
2696
+ "learning_rate": 2.676853546260791e-06,
2697
+ "loss": 0.8789139986038208,
2698
+ "mean_token_accuracy": 0.727929525077343,
2699
+ "num_tokens": 25958833.0,
2700
+ "step": 269
2701
+ },
2702
+ {
2703
+ "entropy": 0.8670627027750015,
2704
+ "epoch": 2.6004842615012107,
2705
+ "grad_norm": 0.163005530834198,
2706
+ "learning_rate": 2.558657276420526e-06,
2707
+ "loss": 0.8627065420150757,
2708
+ "mean_token_accuracy": 0.7318699434399605,
2709
+ "num_tokens": 26056918.0,
2710
+ "step": 270
2711
+ },
2712
+ {
2713
+ "entropy": 0.908613532781601,
2714
+ "epoch": 2.610169491525424,
2715
+ "grad_norm": 0.17326155304908752,
2716
+ "learning_rate": 2.442988913535374e-06,
2717
+ "loss": 0.9048829078674316,
2718
+ "mean_token_accuracy": 0.7207196801900864,
2719
+ "num_tokens": 26154451.0,
2720
+ "step": 271
2721
+ },
2722
+ {
2723
+ "entropy": 0.8706764653325081,
2724
+ "epoch": 2.619854721549637,
2725
+ "grad_norm": 0.16498690843582153,
2726
+ "learning_rate": 2.3298614870778835e-06,
2727
+ "loss": 0.8654090762138367,
2728
+ "mean_token_accuracy": 0.7292862460017204,
2729
+ "num_tokens": 26251471.0,
2730
+ "step": 272
2731
+ },
2732
+ {
2733
+ "entropy": 0.8678311705589294,
2734
+ "epoch": 2.62953995157385,
2735
+ "grad_norm": 0.1647648960351944,
2736
+ "learning_rate": 2.219287740296605e-06,
2737
+ "loss": 0.8630282282829285,
2738
+ "mean_token_accuracy": 0.7309604361653328,
2739
+ "num_tokens": 26348513.0,
2740
+ "step": 273
2741
+ },
2742
+ {
2743
+ "entropy": 0.880488395690918,
2744
+ "epoch": 2.639225181598063,
2745
+ "grad_norm": 0.16676588356494904,
2746
+ "learning_rate": 2.1112801287806378e-06,
2747
+ "loss": 0.8745177984237671,
2748
+ "mean_token_accuracy": 0.7281879782676697,
2749
+ "num_tokens": 26445471.0,
2750
+ "step": 274
2751
+ },
2752
+ {
2753
+ "entropy": 0.8840675577521324,
2754
+ "epoch": 2.648910411622276,
2755
+ "grad_norm": 0.16714756190776825,
2756
+ "learning_rate": 2.0058508190565315e-06,
2757
+ "loss": 0.8826640844345093,
2758
+ "mean_token_accuracy": 0.7246443182229996,
2759
+ "num_tokens": 26543156.0,
2760
+ "step": 275
2761
+ },
2762
+ {
2763
+ "entropy": 0.8701118975877762,
2764
+ "epoch": 2.658595641646489,
2765
+ "grad_norm": 0.1623620241880417,
2766
+ "learning_rate": 1.9030116872178316e-06,
2767
+ "loss": 0.8630843162536621,
2768
+ "mean_token_accuracy": 0.7315845564007759,
2769
+ "num_tokens": 26640670.0,
2770
+ "step": 276
2771
+ },
2772
+ {
2773
+ "entropy": 0.860944889485836,
2774
+ "epoch": 2.668280871670702,
2775
+ "grad_norm": 0.1667710393667221,
2776
+ "learning_rate": 1.8027743175872664e-06,
2777
+ "loss": 0.8536238074302673,
2778
+ "mean_token_accuracy": 0.7340292409062386,
2779
+ "num_tokens": 26736855.0,
2780
+ "step": 277
2781
+ },
2782
+ {
2783
+ "entropy": 0.85859464854002,
2784
+ "epoch": 2.6779661016949152,
2785
+ "grad_norm": 0.16684262454509735,
2786
+ "learning_rate": 1.7051500014118454e-06,
2787
+ "loss": 0.8532140254974365,
2788
+ "mean_token_accuracy": 0.7341034114360809,
2789
+ "num_tokens": 26832927.0,
2790
+ "step": 278
2791
+ },
2792
+ {
2793
+ "entropy": 0.866545133292675,
2794
+ "epoch": 2.6876513317191284,
2795
+ "grad_norm": 0.16754300892353058,
2796
+ "learning_rate": 1.610149735590949e-06,
2797
+ "loss": 0.861035943031311,
2798
+ "mean_token_accuracy": 0.7316532135009766,
2799
+ "num_tokens": 26929935.0,
2800
+ "step": 279
2801
+ },
2802
+ {
2803
+ "entropy": 0.8929876312613487,
2804
+ "epoch": 2.6973365617433416,
2805
+ "grad_norm": 0.17306716740131378,
2806
+ "learning_rate": 1.517784221437568e-06,
2807
+ "loss": 0.8875805735588074,
2808
+ "mean_token_accuracy": 0.724287211894989,
2809
+ "num_tokens": 27027073.0,
2810
+ "step": 280
2811
+ },
2812
+ {
2813
+ "entropy": 0.8867751955986023,
2814
+ "epoch": 2.7070217917675548,
2815
+ "grad_norm": 0.16279691457748413,
2816
+ "learning_rate": 1.428063863472895e-06,
2817
+ "loss": 0.8748654723167419,
2818
+ "mean_token_accuracy": 0.7275810539722443,
2819
+ "num_tokens": 27124636.0,
2820
+ "step": 281
2821
+ },
2822
+ {
2823
+ "entropy": 0.8712080791592598,
2824
+ "epoch": 2.7167070217917675,
2825
+ "grad_norm": 0.16669721901416779,
2826
+ "learning_rate": 1.3409987682542619e-06,
2827
+ "loss": 0.8634123802185059,
2828
+ "mean_token_accuracy": 0.7320047318935394,
2829
+ "num_tokens": 27221555.0,
2830
+ "step": 282
2831
+ },
2832
+ {
2833
+ "entropy": 0.8786909580230713,
2834
+ "epoch": 2.7263922518159807,
2835
+ "grad_norm": 0.1684892624616623,
2836
+ "learning_rate": 1.2565987432367033e-06,
2837
+ "loss": 0.8696650862693787,
2838
+ "mean_token_accuracy": 0.7279440686106682,
2839
+ "num_tokens": 27319178.0,
2840
+ "step": 283
2841
+ },
2842
+ {
2843
+ "entropy": 0.9131262972950935,
2844
+ "epoch": 2.736077481840194,
2845
+ "grad_norm": 0.16781920194625854,
2846
+ "learning_rate": 1.1748732956682025e-06,
2847
+ "loss": 0.9089131355285645,
2848
+ "mean_token_accuracy": 0.7195497453212738,
2849
+ "num_tokens": 27415717.0,
2850
+ "step": 284
2851
+ },
2852
+ {
2853
+ "entropy": 0.8785578608512878,
2854
+ "epoch": 2.7457627118644066,
2855
+ "grad_norm": 0.16821780800819397,
2856
+ "learning_rate": 1.095831631518729e-06,
2857
+ "loss": 0.8705630302429199,
2858
+ "mean_token_accuracy": 0.728852316737175,
2859
+ "num_tokens": 27511588.0,
2860
+ "step": 285
2861
+ },
2862
+ {
2863
+ "entropy": 0.8518342152237892,
2864
+ "epoch": 2.7554479418886197,
2865
+ "grad_norm": 0.16426897048950195,
2866
+ "learning_rate": 1.0194826544432518e-06,
2867
+ "loss": 0.8457784056663513,
2868
+ "mean_token_accuracy": 0.7356671541929245,
2869
+ "num_tokens": 27608214.0,
2870
+ "step": 286
2871
+ },
2872
+ {
2873
+ "entropy": 0.8646640926599503,
2874
+ "epoch": 2.765133171912833,
2875
+ "grad_norm": 0.17445868253707886,
2876
+ "learning_rate": 9.458349647787662e-07,
2877
+ "loss": 0.8553757667541504,
2878
+ "mean_token_accuracy": 0.7336735799908638,
2879
+ "num_tokens": 27705368.0,
2880
+ "step": 287
2881
+ },
2882
+ {
2883
+ "entropy": 0.9048518687486649,
2884
+ "epoch": 2.774818401937046,
2885
+ "grad_norm": 0.29988086223602295,
2886
+ "learning_rate": 8.748968585755274e-07,
2887
+ "loss": 0.8900648951530457,
2888
+ "mean_token_accuracy": 0.7211423367261887,
2889
+ "num_tokens": 27799157.0,
2890
+ "step": 288
2891
+ },
2892
+ {
2893
+ "entropy": 0.889445461332798,
2894
+ "epoch": 2.7845036319612593,
2895
+ "grad_norm": 0.16926804184913635,
2896
+ "learning_rate": 8.066763266625282e-07,
2897
+ "loss": 0.883939266204834,
2898
+ "mean_token_accuracy": 0.7246285825967789,
2899
+ "num_tokens": 27896927.0,
2900
+ "step": 289
2901
+ },
2902
+ {
2903
+ "entropy": 0.8714011684060097,
2904
+ "epoch": 2.794188861985472,
2905
+ "grad_norm": 0.16537313163280487,
2906
+ "learning_rate": 7.411810537473752e-07,
2907
+ "loss": 0.8596811294555664,
2908
+ "mean_token_accuracy": 0.7307524606585503,
2909
+ "num_tokens": 27994705.0,
2910
+ "step": 290
2911
+ },
2912
+ {
2913
+ "entropy": 0.8982528895139694,
2914
+ "epoch": 2.803874092009685,
2915
+ "grad_norm": 0.1691625565290451,
2916
+ "learning_rate": 6.784184175506358e-07,
2917
+ "loss": 0.8857760429382324,
2918
+ "mean_token_accuracy": 0.7251979038119316,
2919
+ "num_tokens": 28091482.0,
2920
+ "step": 291
2921
+ },
2922
+ {
2923
+ "entropy": 0.8777411133050919,
2924
+ "epoch": 2.8135593220338984,
2925
+ "grad_norm": 0.1643359661102295,
2926
+ "learning_rate": 6.18395487974796e-07,
2927
+ "loss": 0.8692206144332886,
2928
+ "mean_token_accuracy": 0.7287580892443657,
2929
+ "num_tokens": 28189404.0,
2930
+ "step": 292
2931
+ },
2932
+ {
2933
+ "entropy": 0.8909460008144379,
2934
+ "epoch": 2.8232445520581115,
2935
+ "grad_norm": 0.1721295416355133,
2936
+ "learning_rate": 5.611190263078464e-07,
2937
+ "loss": 0.8800960779190063,
2938
+ "mean_token_accuracy": 0.7274835258722305,
2939
+ "num_tokens": 28284013.0,
2940
+ "step": 293
2941
+ },
2942
+ {
2943
+ "entropy": 0.9165419265627861,
2944
+ "epoch": 2.8329297820823243,
2945
+ "grad_norm": 0.16794772446155548,
2946
+ "learning_rate": 5.065954844616721e-07,
2947
+ "loss": 0.9082227945327759,
2948
+ "mean_token_accuracy": 0.7175077274441719,
2949
+ "num_tokens": 28381135.0,
2950
+ "step": 294
2951
+ },
2952
+ {
2953
+ "entropy": 0.9155106991529465,
2954
+ "epoch": 2.8426150121065374,
2955
+ "grad_norm": 0.1717296838760376,
2956
+ "learning_rate": 4.548310042452736e-07,
2957
+ "loss": 0.9060653448104858,
2958
+ "mean_token_accuracy": 0.7178425192832947,
2959
+ "num_tokens": 28477566.0,
2960
+ "step": 295
2961
+ },
2962
+ {
2963
+ "entropy": 0.8979469537734985,
2964
+ "epoch": 2.8523002421307506,
2965
+ "grad_norm": 0.162934347987175,
2966
+ "learning_rate": 4.05831416672911e-07,
2967
+ "loss": 0.8928146362304688,
2968
+ "mean_token_accuracy": 0.724173940718174,
2969
+ "num_tokens": 28575460.0,
2970
+ "step": 296
2971
+ },
2972
+ {
2973
+ "entropy": 0.8660886436700821,
2974
+ "epoch": 2.861985472154964,
2975
+ "grad_norm": 0.1685783714056015,
2976
+ "learning_rate": 3.5960224130728857e-07,
2977
+ "loss": 0.8615542650222778,
2978
+ "mean_token_accuracy": 0.7313328757882118,
2979
+ "num_tokens": 28672164.0,
2980
+ "step": 297
2981
+ },
2982
+ {
2983
+ "entropy": 0.8817431479692459,
2984
+ "epoch": 2.871670702179177,
2985
+ "grad_norm": 0.16239479184150696,
2986
+ "learning_rate": 3.1614868563778487e-07,
2987
+ "loss": 0.8701106905937195,
2988
+ "mean_token_accuracy": 0.7281669601798058,
2989
+ "num_tokens": 28769774.0,
2990
+ "step": 298
2991
+ },
2992
+ {
2993
+ "entropy": 0.8943187296390533,
2994
+ "epoch": 2.8813559322033897,
2995
+ "grad_norm": 0.16934892535209656,
2996
+ "learning_rate": 2.7547564449386664e-07,
2997
+ "loss": 0.8885952830314636,
2998
+ "mean_token_accuracy": 0.7244044989347458,
2999
+ "num_tokens": 28866772.0,
3000
+ "step": 299
3001
+ },
3002
+ {
3003
+ "entropy": 0.8902676627039909,
3004
+ "epoch": 2.891041162227603,
3005
+ "grad_norm": 0.16362088918685913,
3006
+ "learning_rate": 2.3758769949370762e-07,
3007
+ "loss": 0.8825098872184753,
3008
+ "mean_token_accuracy": 0.7249467596411705,
3009
+ "num_tokens": 28964603.0,
3010
+ "step": 300
3011
+ },
3012
+ {
3013
+ "entropy": 0.8681572675704956,
3014
+ "epoch": 2.900726392251816,
3015
+ "grad_norm": 0.16239318251609802,
3016
+ "learning_rate": 2.0248911852807916e-07,
3017
+ "loss": 0.8635715246200562,
3018
+ "mean_token_accuracy": 0.7294866740703583,
3019
+ "num_tokens": 29061694.0,
3020
+ "step": 301
3021
+ },
3022
+ {
3023
+ "entropy": 0.8866652101278305,
3024
+ "epoch": 2.910411622276029,
3025
+ "grad_norm": 0.17414243519306183,
3026
+ "learning_rate": 1.7018385527961533e-07,
3027
+ "loss": 0.8773064613342285,
3028
+ "mean_token_accuracy": 0.7276343405246735,
3029
+ "num_tokens": 29159530.0,
3030
+ "step": 302
3031
+ },
3032
+ {
3033
+ "entropy": 0.8873912766575813,
3034
+ "epoch": 2.920096852300242,
3035
+ "grad_norm": 0.1680562049150467,
3036
+ "learning_rate": 1.406755487774386e-07,
3037
+ "loss": 0.8811909556388855,
3038
+ "mean_token_accuracy": 0.7253095731139183,
3039
+ "num_tokens": 29256889.0,
3040
+ "step": 303
3041
+ },
3042
+ {
3043
+ "entropy": 0.8731345906853676,
3044
+ "epoch": 2.929782082324455,
3045
+ "grad_norm": 0.16301023960113525,
3046
+ "learning_rate": 1.1396752298723501e-07,
3047
+ "loss": 0.8583449721336365,
3048
+ "mean_token_accuracy": 0.7309349551796913,
3049
+ "num_tokens": 29352923.0,
3050
+ "step": 304
3051
+ },
3052
+ {
3053
+ "entropy": 0.8695443421602249,
3054
+ "epoch": 2.9394673123486683,
3055
+ "grad_norm": 0.16419994831085205,
3056
+ "learning_rate": 9.006278643683696e-08,
3057
+ "loss": 0.860053539276123,
3058
+ "mean_token_accuracy": 0.7292449474334717,
3059
+ "num_tokens": 29450406.0,
3060
+ "step": 305
3061
+ },
3062
+ {
3063
+ "entropy": 0.8868359550833702,
3064
+ "epoch": 2.9491525423728815,
3065
+ "grad_norm": 0.16607250273227692,
3066
+ "learning_rate": 6.896403187732215e-08,
3067
+ "loss": 0.8776463866233826,
3068
+ "mean_token_accuracy": 0.7254452556371689,
3069
+ "num_tokens": 29546450.0,
3070
+ "step": 306
3071
+ },
3072
+ {
3073
+ "entropy": 0.922260083258152,
3074
+ "epoch": 2.9588377723970947,
3075
+ "grad_norm": 0.16734899580478668,
3076
+ "learning_rate": 5.0673635979686665e-08,
3077
+ "loss": 0.9174036979675293,
3078
+ "mean_token_accuracy": 0.7150839567184448,
3079
+ "num_tokens": 29644332.0,
3080
+ "step": 307
3081
+ },
3082
+ {
3083
+ "entropy": 0.88863155990839,
3084
+ "epoch": 2.9685230024213074,
3085
+ "grad_norm": 0.16869144141674042,
3086
+ "learning_rate": 3.5193659067131455e-08,
3087
+ "loss": 0.8817811012268066,
3088
+ "mean_token_accuracy": 0.7255235239863396,
3089
+ "num_tokens": 29741107.0,
3090
+ "step": 308
3091
+ },
3092
+ {
3093
+ "entropy": 0.8886986747384071,
3094
+ "epoch": 2.9782082324455206,
3095
+ "grad_norm": 0.16776297986507416,
3096
+ "learning_rate": 2.2525844882964607e-08,
3097
+ "loss": 0.8872591853141785,
3098
+ "mean_token_accuracy": 0.7270913049578667,
3099
+ "num_tokens": 29837838.0,
3100
+ "step": 309
3101
+ },
3102
+ {
3103
+ "entropy": 0.8665271773934364,
3104
+ "epoch": 2.9878934624697338,
3105
+ "grad_norm": 0.16298924386501312,
3106
+ "learning_rate": 1.267162039418901e-08,
3107
+ "loss": 0.8613755702972412,
3108
+ "mean_token_accuracy": 0.7298174202442169,
3109
+ "num_tokens": 29935446.0,
3110
+ "step": 310
3111
+ },
3112
+ {
3113
+ "entropy": 0.8940041065216064,
3114
+ "epoch": 2.9975786924939465,
3115
+ "grad_norm": 0.1684512197971344,
3116
+ "learning_rate": 5.632095630764278e-09,
3117
+ "loss": 0.886758029460907,
3118
+ "mean_token_accuracy": 0.7247171625494957,
3119
+ "num_tokens": 30032365.0,
3120
+ "step": 311
3121
+ },
3122
+ {
3123
+ "entropy": 0.8987793922424316,
3124
+ "epoch": 3.0,
3125
+ "grad_norm": 0.33889657258987427,
3126
+ "learning_rate": 1.4080635605512316e-09,
3127
+ "loss": 0.8976651430130005,
3128
+ "mean_token_accuracy": 0.723215252161026,
3129
+ "num_tokens": 30056037.0,
3130
+ "step": 312
3131
+ }
3132
+ ],
3133
+ "logging_steps": 1,
3134
+ "max_steps": 312,
3135
+ "num_input_tokens_seen": 0,
3136
+ "num_train_epochs": 3,
3137
+ "save_steps": 500,
3138
+ "stateful_callbacks": {
3139
+ "TrainerControl": {
3140
+ "args": {
3141
+ "should_epoch_stop": false,
3142
+ "should_evaluate": false,
3143
+ "should_log": false,
3144
+ "should_save": true,
3145
+ "should_training_stop": true
3146
+ },
3147
+ "attributes": {}
3148
+ }
3149
+ },
3150
+ "total_flos": 4.2277901720847647e+18,
3151
+ "train_batch_size": 1,
3152
+ "trial_name": null,
3153
+ "trial_params": null
3154
+ }