avreymi commited on
Commit
68cb86a
·
verified ·
1 Parent(s): 030bf59

Training in progress, step 293, checkpoint

Browse files
checkpoint-293/README.md ADDED
@@ -0,0 +1,209 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: dicta-il/dictalm2.0-instruct
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:dicta-il/dictalm2.0-instruct
7
+ - lora
8
+ - sft
9
+ - transformers
10
+ - trl
11
+ ---
12
+
13
+ # Model Card for Model ID
14
+
15
+ <!-- Provide a quick summary of what the model is/does. -->
16
+
17
+
18
+
19
+ ## Model Details
20
+
21
+ ### Model Description
22
+
23
+ <!-- Provide a longer summary of what this model is. -->
24
+
25
+
26
+
27
+ - **Developed by:** [More Information Needed]
28
+ - **Funded by [optional]:** [More Information Needed]
29
+ - **Shared by [optional]:** [More Information Needed]
30
+ - **Model type:** [More Information Needed]
31
+ - **Language(s) (NLP):** [More Information Needed]
32
+ - **License:** [More Information Needed]
33
+ - **Finetuned from model [optional]:** [More Information Needed]
34
+
35
+ ### Model Sources [optional]
36
+
37
+ <!-- Provide the basic links for the model. -->
38
+
39
+ - **Repository:** [More Information Needed]
40
+ - **Paper [optional]:** [More Information Needed]
41
+ - **Demo [optional]:** [More Information Needed]
42
+
43
+ ## Uses
44
+
45
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
46
+
47
+ ### Direct Use
48
+
49
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
50
+
51
+ [More Information Needed]
52
+
53
+ ### Downstream Use [optional]
54
+
55
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
56
+
57
+ [More Information Needed]
58
+
59
+ ### Out-of-Scope Use
60
+
61
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
62
+
63
+ [More Information Needed]
64
+
65
+ ## Bias, Risks, and Limitations
66
+
67
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
68
+
69
+ [More Information Needed]
70
+
71
+ ### Recommendations
72
+
73
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
74
+
75
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
76
+
77
+ ## How to Get Started with the Model
78
+
79
+ Use the code below to get started with the model.
80
+
81
+ [More Information Needed]
82
+
83
+ ## Training Details
84
+
85
+ ### Training Data
86
+
87
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
88
+
89
+ [More Information Needed]
90
+
91
+ ### Training Procedure
92
+
93
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
94
+
95
+ #### Preprocessing [optional]
96
+
97
+ [More Information Needed]
98
+
99
+
100
+ #### Training Hyperparameters
101
+
102
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
103
+
104
+ #### Speeds, Sizes, Times [optional]
105
+
106
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
107
+
108
+ [More Information Needed]
109
+
110
+ ## Evaluation
111
+
112
+ <!-- This section describes the evaluation protocols and provides the results. -->
113
+
114
+ ### Testing Data, Factors & Metrics
115
+
116
+ #### Testing Data
117
+
118
+ <!-- This should link to a Dataset Card if possible. -->
119
+
120
+ [More Information Needed]
121
+
122
+ #### Factors
123
+
124
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
125
+
126
+ [More Information Needed]
127
+
128
+ #### Metrics
129
+
130
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
131
+
132
+ [More Information Needed]
133
+
134
+ ### Results
135
+
136
+ [More Information Needed]
137
+
138
+ #### Summary
139
+
140
+
141
+
142
+ ## Model Examination [optional]
143
+
144
+ <!-- Relevant interpretability work for the model goes here -->
145
+
146
+ [More Information Needed]
147
+
148
+ ## Environmental Impact
149
+
150
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
151
+
152
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
153
+
154
+ - **Hardware Type:** [More Information Needed]
155
+ - **Hours used:** [More Information Needed]
156
+ - **Cloud Provider:** [More Information Needed]
157
+ - **Compute Region:** [More Information Needed]
158
+ - **Carbon Emitted:** [More Information Needed]
159
+
160
+ ## Technical Specifications [optional]
161
+
162
+ ### Model Architecture and Objective
163
+
164
+ [More Information Needed]
165
+
166
+ ### Compute Infrastructure
167
+
168
+ [More Information Needed]
169
+
170
+ #### Hardware
171
+
172
+ [More Information Needed]
173
+
174
+ #### Software
175
+
176
+ [More Information Needed]
177
+
178
+ ## Citation [optional]
179
+
180
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
181
+
182
+ **BibTeX:**
183
+
184
+ [More Information Needed]
185
+
186
+ **APA:**
187
+
188
+ [More Information Needed]
189
+
190
+ ## Glossary [optional]
191
+
192
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
193
+
194
+ [More Information Needed]
195
+
196
+ ## More Information [optional]
197
+
198
+ [More Information Needed]
199
+
200
+ ## Model Card Authors [optional]
201
+
202
+ [More Information Needed]
203
+
204
+ ## Model Card Contact
205
+
206
+ [More Information Needed]
207
+ ### Framework versions
208
+
209
+ - PEFT 0.20.0
checkpoint-293/adapter_config.json ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alora_invocation_tokens": null,
3
+ "alpha_pattern": {},
4
+ "arrow_config": null,
5
+ "auto_mapping": null,
6
+ "base_model_name_or_path": "dicta-il/dictalm2.0-instruct",
7
+ "bias": "none",
8
+ "corda_config": null,
9
+ "ensure_weight_tying": false,
10
+ "eva_config": null,
11
+ "exclude_modules": null,
12
+ "fan_in_fan_out": false,
13
+ "inference_mode": true,
14
+ "init_lora_weights": true,
15
+ "layer_replication": null,
16
+ "layers_pattern": null,
17
+ "layers_to_transform": null,
18
+ "loftq_config": {},
19
+ "lora_alpha": 64,
20
+ "lora_bias": false,
21
+ "lora_dropout": 0.05,
22
+ "lora_ga_config": null,
23
+ "megatron_config": null,
24
+ "megatron_core": "megatron.core",
25
+ "modules_to_save": null,
26
+ "monteclora_config": null,
27
+ "peft_type": "LORA",
28
+ "peft_version": "0.20.0",
29
+ "qalora_group_size": 16,
30
+ "r": 32,
31
+ "rank_pattern": {},
32
+ "revision": null,
33
+ "target_modules": [
34
+ "up_proj",
35
+ "down_proj",
36
+ "v_proj",
37
+ "o_proj",
38
+ "gate_proj",
39
+ "q_proj",
40
+ "k_proj"
41
+ ],
42
+ "target_parameters": null,
43
+ "task_type": "CAUSAL_LM",
44
+ "trainable_token_indices": null,
45
+ "use_bdlora": null,
46
+ "use_dora": false,
47
+ "use_qalora": false,
48
+ "use_rslora": false,
49
+ "velora_config": null
50
+ }
checkpoint-293/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76fe8326d0efed6f14f19ae9eb1a5bd56a80df133748a5bfb12b6710a147dda4
3
+ size 335604696
checkpoint-293/chat_template.jinja ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ {{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]
2
+ ' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}
checkpoint-293/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:25cd918058276f44c5d507e031d72a861594b9efb432efbc20fb560194df961f
3
+ size 671473443
checkpoint-293/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b108bf6f1c7c1782af50cc132fe8aff310f3bedbeb6196c1ded2c4c47ab58928
3
+ size 14645
checkpoint-293/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a34ef5a29cb80b88f6231871c7af2e6d36f482d42394c64793e134bee25f4355
3
+ size 1465
checkpoint-293/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
checkpoint-293/tokenizer_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<s>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "</s>",
6
+ "is_local": false,
7
+ "legacy": true,
8
+ "local_files_only": false,
9
+ "model_max_length": 1000000000000000019884624838656,
10
+ "pad_token": "</s>",
11
+ "sp_model_kwargs": {},
12
+ "spaces_between_special_tokens": false,
13
+ "tokenizer_class": "TokenizersBackend",
14
+ "unk_token": "<unk>",
15
+ "use_default_system_prompt": false
16
+ }
checkpoint-293/trainer_state.json ADDED
@@ -0,0 +1,390 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": 293,
3
+ "best_metric": 0.9774845838546753,
4
+ "best_model_checkpoint": "/data/output/checkpoint-293",
5
+ "epoch": 1.0,
6
+ "eval_steps": 50,
7
+ "global_step": 293,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "entropy": 1.1713112026453019,
14
+ "epoch": 0.034158838599487616,
15
+ "grad_norm": 3.3329014778137207,
16
+ "learning_rate": 0.00012,
17
+ "loss": 1.2999439239501953,
18
+ "mean_token_accuracy": 0.6959523037075996,
19
+ "num_tokens": 643010.0,
20
+ "step": 10
21
+ },
22
+ {
23
+ "entropy": 1.1543982975184917,
24
+ "epoch": 0.06831767719897523,
25
+ "grad_norm": 2.581118106842041,
26
+ "learning_rate": 0.00019989785286500295,
27
+ "loss": 1.1518776893615723,
28
+ "mean_token_accuracy": 0.7155417971313,
29
+ "num_tokens": 1251488.0,
30
+ "step": 20
31
+ },
32
+ {
33
+ "entropy": 0.9459363672882318,
34
+ "epoch": 0.10247651579846286,
35
+ "grad_norm": 2.3779942989349365,
36
+ "learning_rate": 0.00019875109281794825,
37
+ "loss": 0.9381013870239258,
38
+ "mean_token_accuracy": 0.7682368114590645,
39
+ "num_tokens": 1696074.0,
40
+ "step": 30
41
+ },
42
+ {
43
+ "entropy": 0.9385578982532025,
44
+ "epoch": 0.13663535439795046,
45
+ "grad_norm": 2.0882723331451416,
46
+ "learning_rate": 0.00019634456691705702,
47
+ "loss": 0.9547281265258789,
48
+ "mean_token_accuracy": 0.7626850850880146,
49
+ "num_tokens": 2006643.0,
50
+ "step": 40
51
+ },
52
+ {
53
+ "entropy": 0.9006230250000954,
54
+ "epoch": 0.1707941929974381,
55
+ "grad_norm": 2.151762008666992,
56
+ "learning_rate": 0.00019270897516847403,
57
+ "loss": 0.9329019546508789,
58
+ "mean_token_accuracy": 0.7718432284891605,
59
+ "num_tokens": 2177826.0,
60
+ "step": 50
61
+ },
62
+ {
63
+ "epoch": 0.1707941929974381,
64
+ "eval_entropy": 1.0149640817940235,
65
+ "eval_loss": 1.071570634841919,
66
+ "eval_mean_token_accuracy": 0.742358745932579,
67
+ "eval_num_tokens": 2177826.0,
68
+ "eval_runtime": 95.085,
69
+ "eval_samples_per_second": 1.052,
70
+ "eval_steps_per_second": 1.052,
71
+ "step": 50
72
+ },
73
+ {
74
+ "entropy": 1.2094877127557992,
75
+ "epoch": 0.2049530315969257,
76
+ "grad_norm": 2.767357587814331,
77
+ "learning_rate": 0.0001878906967484966,
78
+ "loss": 1.209201717376709,
79
+ "mean_token_accuracy": 0.7017950341105461,
80
+ "num_tokens": 644405.0,
81
+ "step": 60
82
+ },
83
+ {
84
+ "entropy": 1.1481261406093837,
85
+ "epoch": 0.23911187019641333,
86
+ "grad_norm": 2.805593252182007,
87
+ "learning_rate": 0.00018195119834528534,
88
+ "loss": 1.1588325500488281,
89
+ "mean_token_accuracy": 0.7221003979444504,
90
+ "num_tokens": 1268539.0,
91
+ "step": 70
92
+ },
93
+ {
94
+ "entropy": 0.9842961743474007,
95
+ "epoch": 0.27327070879590093,
96
+ "grad_norm": 2.233400344848633,
97
+ "learning_rate": 0.000174966250029467,
98
+ "loss": 1.0100913047790527,
99
+ "mean_token_accuracy": 0.7561626233160496,
100
+ "num_tokens": 1729123.0,
101
+ "step": 80
102
+ },
103
+ {
104
+ "entropy": 0.9889072403311729,
105
+ "epoch": 0.3074295473953886,
106
+ "grad_norm": 1.992038369178772,
107
+ "learning_rate": 0.0001670249586567531,
108
+ "loss": 1.0143807411193848,
109
+ "mean_token_accuracy": 0.7507779069244862,
110
+ "num_tokens": 2034736.0,
111
+ "step": 90
112
+ },
113
+ {
114
+ "entropy": 0.8992462156340479,
115
+ "epoch": 0.3415883859948762,
116
+ "grad_norm": 2.0321593284606934,
117
+ "learning_rate": 0.00015822863113343935,
118
+ "loss": 0.9014181137084961,
119
+ "mean_token_accuracy": 0.7751547530293464,
120
+ "num_tokens": 2199862.0,
121
+ "step": 100
122
+ },
123
+ {
124
+ "epoch": 0.3415883859948762,
125
+ "eval_entropy": 0.9943241326510907,
126
+ "eval_loss": 1.043296456336975,
127
+ "eval_mean_token_accuracy": 0.7519984298944473,
128
+ "eval_num_tokens": 2199862.0,
129
+ "eval_runtime": 95.5899,
130
+ "eval_samples_per_second": 1.046,
131
+ "eval_steps_per_second": 1.046,
132
+ "step": 100
133
+ },
134
+ {
135
+ "entropy": 1.1760564662516118,
136
+ "epoch": 0.37574722459436377,
137
+ "grad_norm": 3.0291037559509277,
138
+ "learning_rate": 0.000148689482046087,
139
+ "loss": 1.2000227928161622,
140
+ "mean_token_accuracy": 0.711453115195036,
141
+ "num_tokens": 643606.0,
142
+ "step": 110
143
+ },
144
+ {
145
+ "entropy": 1.1263522632420062,
146
+ "epoch": 0.4099060631938514,
147
+ "grad_norm": 2.554089307785034,
148
+ "learning_rate": 0.00013852920214212964,
149
+ "loss": 1.1538644790649415,
150
+ "mean_token_accuracy": 0.7203516490757466,
151
+ "num_tokens": 1250932.0,
152
+ "step": 120
153
+ },
154
+ {
155
+ "entropy": 0.9661639135330915,
156
+ "epoch": 0.444064901793339,
157
+ "grad_norm": 2.2706644535064697,
158
+ "learning_rate": 0.0001278774059232723,
159
+ "loss": 0.9473760604858399,
160
+ "mean_token_accuracy": 0.7646104976534843,
161
+ "num_tokens": 1690510.0,
162
+ "step": 130
163
+ },
164
+ {
165
+ "entropy": 0.9644285907968879,
166
+ "epoch": 0.47822374039282667,
167
+ "grad_norm": 1.8326361179351807,
168
+ "learning_rate": 0.00011686997815570473,
169
+ "loss": 0.9465847969055176,
170
+ "mean_token_accuracy": 0.760813869535923,
171
+ "num_tokens": 1991606.0,
172
+ "step": 140
173
+ },
174
+ {
175
+ "entropy": 0.847778935264796,
176
+ "epoch": 0.5123825789923142,
177
+ "grad_norm": 2.5428972244262695,
178
+ "learning_rate": 0.00010564734039066699,
179
+ "loss": 0.8805885314941406,
180
+ "mean_token_accuracy": 0.784095474332571,
181
+ "num_tokens": 2153462.0,
182
+ "step": 150
183
+ },
184
+ {
185
+ "epoch": 0.5123825789923142,
186
+ "eval_entropy": 1.0146987434476613,
187
+ "eval_loss": 1.0139193534851074,
188
+ "eval_mean_token_accuracy": 0.7562771487236023,
189
+ "eval_num_tokens": 2153462.0,
190
+ "eval_runtime": 95.2112,
191
+ "eval_samples_per_second": 1.05,
192
+ "eval_steps_per_second": 1.05,
193
+ "step": 150
194
+ },
195
+ {
196
+ "entropy": 1.2566933140158654,
197
+ "epoch": 0.5465414175918019,
198
+ "grad_norm": 2.956472873687744,
199
+ "learning_rate": 9.435265960933302e-05,
200
+ "loss": 1.2643850326538086,
201
+ "mean_token_accuracy": 0.700284918397665,
202
+ "num_tokens": 644011.0,
203
+ "step": 160
204
+ },
205
+ {
206
+ "entropy": 1.1445974234491587,
207
+ "epoch": 0.5807002561912895,
208
+ "grad_norm": 2.522840738296509,
209
+ "learning_rate": 8.313002184429529e-05,
210
+ "loss": 1.1480578422546386,
211
+ "mean_token_accuracy": 0.7231714740395546,
212
+ "num_tokens": 1272288.0,
213
+ "step": 170
214
+ },
215
+ {
216
+ "entropy": 0.9980672411620617,
217
+ "epoch": 0.6148590947907772,
218
+ "grad_norm": 2.023250102996826,
219
+ "learning_rate": 7.21225940767277e-05,
220
+ "loss": 1.0411287307739259,
221
+ "mean_token_accuracy": 0.74666218906641,
222
+ "num_tokens": 1728820.0,
223
+ "step": 180
224
+ },
225
+ {
226
+ "entropy": 0.9762071590870619,
227
+ "epoch": 0.6490179333902647,
228
+ "grad_norm": 1.7645409107208252,
229
+ "learning_rate": 6.147079785787038e-05,
230
+ "loss": 0.9519403457641602,
231
+ "mean_token_accuracy": 0.7596977911889553,
232
+ "num_tokens": 2027911.0,
233
+ "step": 190
234
+ },
235
+ {
236
+ "entropy": 0.829610382579267,
237
+ "epoch": 0.6831767719897524,
238
+ "grad_norm": 1.8310364484786987,
239
+ "learning_rate": 5.131051795391302e-05,
240
+ "loss": 0.8207592964172363,
241
+ "mean_token_accuracy": 0.7889963142573834,
242
+ "num_tokens": 2200169.0,
243
+ "step": 200
244
+ },
245
+ {
246
+ "epoch": 0.6831767719897524,
247
+ "eval_entropy": 1.0059119730442763,
248
+ "eval_loss": 1.002184271812439,
249
+ "eval_mean_token_accuracy": 0.7572826623916626,
250
+ "eval_num_tokens": 2200169.0,
251
+ "eval_runtime": 95.2716,
252
+ "eval_samples_per_second": 1.05,
253
+ "eval_steps_per_second": 1.05,
254
+ "step": 200
255
+ },
256
+ {
257
+ "entropy": 1.1494356069713831,
258
+ "epoch": 0.71733561058924,
259
+ "grad_norm": 3.3939480781555176,
260
+ "learning_rate": 4.1771368866560665e-05,
261
+ "loss": 1.169883918762207,
262
+ "mean_token_accuracy": 0.7144785463809967,
263
+ "num_tokens": 2843646.0,
264
+ "step": 210
265
+ },
266
+ {
267
+ "entropy": 1.1186032563447952,
268
+ "epoch": 0.7514944491887275,
269
+ "grad_norm": 2.491504192352295,
270
+ "learning_rate": 3.2975041343246936e-05,
271
+ "loss": 1.109489631652832,
272
+ "mean_token_accuracy": 0.7285458266735076,
273
+ "num_tokens": 3464241.0,
274
+ "step": 220
275
+ },
276
+ {
277
+ "entropy": 0.9699326818808913,
278
+ "epoch": 0.7856532877882152,
279
+ "grad_norm": 1.8113044500350952,
280
+ "learning_rate": 2.5033749970533015e-05,
281
+ "loss": 0.9755414009094239,
282
+ "mean_token_accuracy": 0.7635111711919308,
283
+ "num_tokens": 3914675.0,
284
+ "step": 230
285
+ },
286
+ {
287
+ "entropy": 0.9966869790107011,
288
+ "epoch": 0.8198121263877028,
289
+ "grad_norm": 1.7850583791732788,
290
+ "learning_rate": 1.8048801654714688e-05,
291
+ "loss": 1.020698356628418,
292
+ "mean_token_accuracy": 0.7484274171292782,
293
+ "num_tokens": 4212723.0,
294
+ "step": 240
295
+ },
296
+ {
297
+ "entropy": 0.8349324990995228,
298
+ "epoch": 0.8539709649871904,
299
+ "grad_norm": 1.9040881395339966,
300
+ "learning_rate": 1.2109303251503434e-05,
301
+ "loss": 0.8317201614379883,
302
+ "mean_token_accuracy": 0.7891084231436253,
303
+ "num_tokens": 4374805.0,
304
+ "step": 250
305
+ },
306
+ {
307
+ "epoch": 0.8539709649871904,
308
+ "eval_entropy": 0.968962878882885,
309
+ "eval_loss": 0.9828886389732361,
310
+ "eval_mean_token_accuracy": 0.7627548807859421,
311
+ "eval_num_tokens": 4374805.0,
312
+ "eval_runtime": 95.2882,
313
+ "eval_samples_per_second": 1.049,
314
+ "eval_steps_per_second": 1.049,
315
+ "step": 250
316
+ },
317
+ {
318
+ "entropy": 1.1486725121736527,
319
+ "epoch": 0.888129803586678,
320
+ "grad_norm": 2.9513657093048096,
321
+ "learning_rate": 7.291024831525961e-06,
322
+ "loss": 1.1687018394470214,
323
+ "mean_token_accuracy": 0.7181367576122284,
324
+ "num_tokens": 5017517.0,
325
+ "step": 260
326
+ },
327
+ {
328
+ "entropy": 1.0854963053017854,
329
+ "epoch": 0.9222886421861657,
330
+ "grad_norm": 1.8715020418167114,
331
+ "learning_rate": 3.655433082942972e-06,
332
+ "loss": 1.1069631576538086,
333
+ "mean_token_accuracy": 0.7262594766914845,
334
+ "num_tokens": 5589315.0,
335
+ "step": 270
336
+ },
337
+ {
338
+ "entropy": 0.8848588610067963,
339
+ "epoch": 0.9564474807856533,
340
+ "grad_norm": 1.542606234550476,
341
+ "learning_rate": 1.2489071820517396e-06,
342
+ "loss": 0.9019373893737793,
343
+ "mean_token_accuracy": 0.7785033553838729,
344
+ "num_tokens": 5963691.0,
345
+ "step": 280
346
+ },
347
+ {
348
+ "entropy": 0.8614852068014442,
349
+ "epoch": 0.9906063193851409,
350
+ "grad_norm": 1.7902292013168335,
351
+ "learning_rate": 1.0214713499706597e-07,
352
+ "loss": 0.843895149230957,
353
+ "mean_token_accuracy": 0.7876522168517113,
354
+ "num_tokens": 6195953.0,
355
+ "step": 290
356
+ },
357
+ {
358
+ "epoch": 1.0,
359
+ "eval_entropy": 0.9732591247558594,
360
+ "eval_loss": 0.9774845838546753,
361
+ "eval_mean_token_accuracy": 0.7655571502447128,
362
+ "eval_num_tokens": 6224558.0,
363
+ "eval_runtime": 95.2553,
364
+ "eval_samples_per_second": 1.05,
365
+ "eval_steps_per_second": 1.05,
366
+ "step": 293
367
+ }
368
+ ],
369
+ "logging_steps": 10,
370
+ "max_steps": 293,
371
+ "num_input_tokens_seen": 0,
372
+ "num_train_epochs": 1,
373
+ "save_steps": 50,
374
+ "stateful_callbacks": {
375
+ "TrainerControl": {
376
+ "args": {
377
+ "should_epoch_stop": false,
378
+ "should_evaluate": false,
379
+ "should_log": false,
380
+ "should_save": true,
381
+ "should_training_stop": true
382
+ },
383
+ "attributes": {}
384
+ }
385
+ },
386
+ "total_flos": 5.5148101705836134e+17,
387
+ "train_batch_size": 2,
388
+ "trial_name": null,
389
+ "trial_params": null
390
+ }
checkpoint-293/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:728565783f40be2b9a04e50a9ae836a7423a70af6895b74d267c7469491832b6
3
+ size 5841