Hugging Face's logo

neurocoder
/
Llama-3.2-1B-Instruct-sft-math-gsm8k

Text Generation
Transformers
TensorBoard
Safetensors
llama
Generated from Trainer
trl
sft
conversational
text-generation-inference
Model card Files
xet
Metrics Community
neurocoder commited on
Commit
14622e1
·
verified ·
1 Parent(s): d6f97c4

Model save

Browse files
Files changed (8) hide show
  1. README.md +58 -0
  2. all_results.json +9 -0
  3. config.json +1 -1
  4. generation_config.json +12 -0
  5. model.safetensors +1 -1
  6. runs/Mar30_11-28-35_a100-f-01/events.out.tfevents.1743294521.a100-f-01 +2 -2
  7. train_results.json +9 -0
  8. trainer_state.json +750 -0
README.md ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: meta-llama/Llama-3.2-1B-Instruct
3
+ library_name: transformers
4
+ model_name: Llama-3.2-1B-Instruct-sft-math-gsm8k
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - sft
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Llama-3.2-1B-Instruct-sft-math-gsm8k
13
+
14
+ This model is a fine-tuned version of [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="neurocoder/Llama-3.2-1B-Instruct-sft-math-gsm8k", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+
31
+
32
+
33
+ This model was trained with SFT.
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.14.0
38
+ - Transformers: 4.50.1
39
+ - Pytorch: 2.5.1
40
+ - Datasets: 3.4.1
41
+ - Tokenizers: 0.21.1
42
+
43
+ ## Citations
44
+
45
+
46
+
47
+ Cite TRL as:
48
+
49
+ ```bibtex
50
+ @misc{vonwerra2022trl,
51
+ title = {{TRL: Transformer Reinforcement Learning}},
52
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
53
+ year = 2020,
54
+ journal = {GitHub repository},
55
+ publisher = {GitHub},
56
+ howpublished = {\url{https://github.com/huggingface/trl}}
57
+ }
58
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 0.9987639060568603,
3
+ "total_flos": 9435624701952000.0,
4
+ "train_loss": 1.491983689412032,
5
+ "train_runtime": 449.9567,
6
+ "train_samples": 7473,
7
+ "train_samples_per_second": 17.977,
8
+ "train_steps_per_second": 1.122
9
+ }
config.json CHANGED
@@ -34,6 +34,6 @@
34
  "tie_word_embeddings": true,
35
  "torch_dtype": "bfloat16",
36
  "transformers_version": "4.50.1",
37
- "use_cache": false,
38
  "vocab_size": 128256
39
  }
 
34
  "tie_word_embeddings": true,
35
  "torch_dtype": "bfloat16",
36
  "transformers_version": "4.50.1",
37
+ "use_cache": true,
38
  "vocab_size": 128256
39
  }
generation_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 128000,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 128001,
6
+ 128008,
7
+ 128009
8
+ ],
9
+ "temperature": 0.6,
10
+ "top_p": 0.9,
11
+ "transformers_version": "4.50.1"
12
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:338545f6e010af19c22f038742d0abad52b9b9d6483d6b4dc6e0bae5dd51475c
3
  size 2471645608
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:067bcde04a17c2d0d1e9cd15bd6038f66c88b34fa176f72285c98c99b068c850
3
  size 2471645608
runs/Mar30_11-28-35_a100-f-01/events.out.tfevents.1743294521.a100-f-01 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4d11171db3c5c31036fdb0b3674c2284d4a946eff78404534ac714499085d2ab
3
- size 22449
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2f9cc53a6fffd95bc902b999758c4ceace2bcc3dfb162e702a8e0a3af94a5c08
3
+ size 27234
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 0.9987639060568603,
3
+ "total_flos": 9435624701952000.0,
4
+ "train_loss": 1.491983689412032,
5
+ "train_runtime": 449.9567,
6
+ "train_samples": 7473,
7
+ "train_samples_per_second": 17.977,
8
+ "train_steps_per_second": 1.122
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,750 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 0.9987639060568603,
6
+ "eval_steps": 500,
7
+ "global_step": 505,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "epoch": 0.009888751545117428,
14
+ "grad_norm": 25.25,
15
+ "learning_rate": 0.0002,
16
+ "loss": 3.4401,
17
+ "step": 5
18
+ },
19
+ {
20
+ "epoch": 0.019777503090234856,
21
+ "grad_norm": 11.125,
22
+ "learning_rate": 0.0002,
23
+ "loss": 2.2116,
24
+ "step": 10
25
+ },
26
+ {
27
+ "epoch": 0.029666254635352288,
28
+ "grad_norm": 4.65625,
29
+ "learning_rate": 0.0002,
30
+ "loss": 1.6815,
31
+ "step": 15
32
+ },
33
+ {
34
+ "epoch": 0.03955500618046971,
35
+ "grad_norm": 13.0,
36
+ "learning_rate": 0.0002,
37
+ "loss": 1.906,
38
+ "step": 20
39
+ },
40
+ {
41
+ "epoch": 0.049443757725587144,
42
+ "grad_norm": 6.03125,
43
+ "learning_rate": 0.0002,
44
+ "loss": 1.8419,
45
+ "step": 25
46
+ },
47
+ {
48
+ "epoch": 0.059332509270704575,
49
+ "grad_norm": 7.125,
50
+ "learning_rate": 0.0002,
51
+ "loss": 1.7867,
52
+ "step": 30
53
+ },
54
+ {
55
+ "epoch": 0.069221260815822,
56
+ "grad_norm": 4.125,
57
+ "learning_rate": 0.0002,
58
+ "loss": 1.7177,
59
+ "step": 35
60
+ },
61
+ {
62
+ "epoch": 0.07911001236093942,
63
+ "grad_norm": 4.375,
64
+ "learning_rate": 0.0002,
65
+ "loss": 1.4636,
66
+ "step": 40
67
+ },
68
+ {
69
+ "epoch": 0.08899876390605686,
70
+ "grad_norm": 42.5,
71
+ "learning_rate": 0.0002,
72
+ "loss": 1.6758,
73
+ "step": 45
74
+ },
75
+ {
76
+ "epoch": 0.09888751545117429,
77
+ "grad_norm": 3.71875,
78
+ "learning_rate": 0.0002,
79
+ "loss": 1.6556,
80
+ "step": 50
81
+ },
82
+ {
83
+ "epoch": 0.10877626699629171,
84
+ "grad_norm": 3.953125,
85
+ "learning_rate": 0.0002,
86
+ "loss": 1.5998,
87
+ "step": 55
88
+ },
89
+ {
90
+ "epoch": 0.11866501854140915,
91
+ "grad_norm": 3.21875,
92
+ "learning_rate": 0.0002,
93
+ "loss": 1.5765,
94
+ "step": 60
95
+ },
96
+ {
97
+ "epoch": 0.12855377008652658,
98
+ "grad_norm": 3.71875,
99
+ "learning_rate": 0.0002,
100
+ "loss": 1.5169,
101
+ "step": 65
102
+ },
103
+ {
104
+ "epoch": 0.138442521631644,
105
+ "grad_norm": 3.359375,
106
+ "learning_rate": 0.0002,
107
+ "loss": 1.582,
108
+ "step": 70
109
+ },
110
+ {
111
+ "epoch": 0.14833127317676142,
112
+ "grad_norm": 4.96875,
113
+ "learning_rate": 0.0002,
114
+ "loss": 1.5751,
115
+ "step": 75
116
+ },
117
+ {
118
+ "epoch": 0.15822002472187885,
119
+ "grad_norm": 3.03125,
120
+ "learning_rate": 0.0002,
121
+ "loss": 1.6266,
122
+ "step": 80
123
+ },
124
+ {
125
+ "epoch": 0.1681087762669963,
126
+ "grad_norm": 3.640625,
127
+ "learning_rate": 0.0002,
128
+ "loss": 1.508,
129
+ "step": 85
130
+ },
131
+ {
132
+ "epoch": 0.17799752781211373,
133
+ "grad_norm": 2.765625,
134
+ "learning_rate": 0.0002,
135
+ "loss": 1.5031,
136
+ "step": 90
137
+ },
138
+ {
139
+ "epoch": 0.18788627935723115,
140
+ "grad_norm": 2.734375,
141
+ "learning_rate": 0.0002,
142
+ "loss": 1.5231,
143
+ "step": 95
144
+ },
145
+ {
146
+ "epoch": 0.19777503090234858,
147
+ "grad_norm": 8.3125,
148
+ "learning_rate": 0.0002,
149
+ "loss": 1.6393,
150
+ "step": 100
151
+ },
152
+ {
153
+ "epoch": 0.207663782447466,
154
+ "grad_norm": 145.0,
155
+ "learning_rate": 0.0002,
156
+ "loss": 1.7596,
157
+ "step": 105
158
+ },
159
+ {
160
+ "epoch": 0.21755253399258342,
161
+ "grad_norm": 2.578125,
162
+ "learning_rate": 0.0002,
163
+ "loss": 1.5453,
164
+ "step": 110
165
+ },
166
+ {
167
+ "epoch": 0.22744128553770088,
168
+ "grad_norm": 3.375,
169
+ "learning_rate": 0.0002,
170
+ "loss": 1.5337,
171
+ "step": 115
172
+ },
173
+ {
174
+ "epoch": 0.2373300370828183,
175
+ "grad_norm": 15.375,
176
+ "learning_rate": 0.0002,
177
+ "loss": 1.6691,
178
+ "step": 120
179
+ },
180
+ {
181
+ "epoch": 0.24721878862793573,
182
+ "grad_norm": 6.5625,
183
+ "learning_rate": 0.0002,
184
+ "loss": 1.5903,
185
+ "step": 125
186
+ },
187
+ {
188
+ "epoch": 0.25710754017305315,
189
+ "grad_norm": 2.953125,
190
+ "learning_rate": 0.0002,
191
+ "loss": 1.4876,
192
+ "step": 130
193
+ },
194
+ {
195
+ "epoch": 0.2669962917181706,
196
+ "grad_norm": 24.875,
197
+ "learning_rate": 0.0002,
198
+ "loss": 1.6292,
199
+ "step": 135
200
+ },
201
+ {
202
+ "epoch": 0.276885043263288,
203
+ "grad_norm": 3.03125,
204
+ "learning_rate": 0.0002,
205
+ "loss": 1.5652,
206
+ "step": 140
207
+ },
208
+ {
209
+ "epoch": 0.2867737948084054,
210
+ "grad_norm": 3.03125,
211
+ "learning_rate": 0.0002,
212
+ "loss": 1.5782,
213
+ "step": 145
214
+ },
215
+ {
216
+ "epoch": 0.29666254635352285,
217
+ "grad_norm": 3.828125,
218
+ "learning_rate": 0.0002,
219
+ "loss": 1.5251,
220
+ "step": 150
221
+ },
222
+ {
223
+ "epoch": 0.3065512978986403,
224
+ "grad_norm": 4.71875,
225
+ "learning_rate": 0.0002,
226
+ "loss": 1.4326,
227
+ "step": 155
228
+ },
229
+ {
230
+ "epoch": 0.3164400494437577,
231
+ "grad_norm": 2.375,
232
+ "learning_rate": 0.0002,
233
+ "loss": 1.5085,
234
+ "step": 160
235
+ },
236
+ {
237
+ "epoch": 0.3263288009888752,
238
+ "grad_norm": 3.84375,
239
+ "learning_rate": 0.0002,
240
+ "loss": 1.482,
241
+ "step": 165
242
+ },
243
+ {
244
+ "epoch": 0.3362175525339926,
245
+ "grad_norm": 2.328125,
246
+ "learning_rate": 0.0002,
247
+ "loss": 1.516,
248
+ "step": 170
249
+ },
250
+ {
251
+ "epoch": 0.34610630407911,
252
+ "grad_norm": 2.453125,
253
+ "learning_rate": 0.0002,
254
+ "loss": 1.5091,
255
+ "step": 175
256
+ },
257
+ {
258
+ "epoch": 0.35599505562422745,
259
+ "grad_norm": 2.625,
260
+ "learning_rate": 0.0002,
261
+ "loss": 1.4517,
262
+ "step": 180
263
+ },
264
+ {
265
+ "epoch": 0.3658838071693449,
266
+ "grad_norm": 2.84375,
267
+ "learning_rate": 0.0002,
268
+ "loss": 1.4647,
269
+ "step": 185
270
+ },
271
+ {
272
+ "epoch": 0.3757725587144623,
273
+ "grad_norm": 2.734375,
274
+ "learning_rate": 0.0002,
275
+ "loss": 1.4764,
276
+ "step": 190
277
+ },
278
+ {
279
+ "epoch": 0.3856613102595797,
280
+ "grad_norm": 2.15625,
281
+ "learning_rate": 0.0002,
282
+ "loss": 1.4254,
283
+ "step": 195
284
+ },
285
+ {
286
+ "epoch": 0.39555006180469715,
287
+ "grad_norm": 2.34375,
288
+ "learning_rate": 0.0002,
289
+ "loss": 1.4868,
290
+ "step": 200
291
+ },
292
+ {
293
+ "epoch": 0.4054388133498146,
294
+ "grad_norm": 2.609375,
295
+ "learning_rate": 0.0002,
296
+ "loss": 1.4534,
297
+ "step": 205
298
+ },
299
+ {
300
+ "epoch": 0.415327564894932,
301
+ "grad_norm": 2.34375,
302
+ "learning_rate": 0.0002,
303
+ "loss": 1.447,
304
+ "step": 210
305
+ },
306
+ {
307
+ "epoch": 0.4252163164400494,
308
+ "grad_norm": 2.3125,
309
+ "learning_rate": 0.0002,
310
+ "loss": 1.4514,
311
+ "step": 215
312
+ },
313
+ {
314
+ "epoch": 0.43510506798516685,
315
+ "grad_norm": 2.109375,
316
+ "learning_rate": 0.0002,
317
+ "loss": 1.4372,
318
+ "step": 220
319
+ },
320
+ {
321
+ "epoch": 0.44499381953028433,
322
+ "grad_norm": 2.859375,
323
+ "learning_rate": 0.0002,
324
+ "loss": 1.3961,
325
+ "step": 225
326
+ },
327
+ {
328
+ "epoch": 0.45488257107540175,
329
+ "grad_norm": 3.0625,
330
+ "learning_rate": 0.0002,
331
+ "loss": 1.4363,
332
+ "step": 230
333
+ },
334
+ {
335
+ "epoch": 0.4647713226205192,
336
+ "grad_norm": 2.703125,
337
+ "learning_rate": 0.0002,
338
+ "loss": 1.4355,
339
+ "step": 235
340
+ },
341
+ {
342
+ "epoch": 0.4746600741656366,
343
+ "grad_norm": 2.8125,
344
+ "learning_rate": 0.0002,
345
+ "loss": 1.525,
346
+ "step": 240
347
+ },
348
+ {
349
+ "epoch": 0.484548825710754,
350
+ "grad_norm": 2.46875,
351
+ "learning_rate": 0.0002,
352
+ "loss": 1.3801,
353
+ "step": 245
354
+ },
355
+ {
356
+ "epoch": 0.49443757725587145,
357
+ "grad_norm": 2.40625,
358
+ "learning_rate": 0.0002,
359
+ "loss": 1.5133,
360
+ "step": 250
361
+ },
362
+ {
363
+ "epoch": 0.5043263288009888,
364
+ "grad_norm": 2.328125,
365
+ "learning_rate": 0.0002,
366
+ "loss": 1.4364,
367
+ "step": 255
368
+ },
369
+ {
370
+ "epoch": 0.5142150803461063,
371
+ "grad_norm": 2.3125,
372
+ "learning_rate": 0.0002,
373
+ "loss": 1.4363,
374
+ "step": 260
375
+ },
376
+ {
377
+ "epoch": 0.5241038318912238,
378
+ "grad_norm": 2.75,
379
+ "learning_rate": 0.0002,
380
+ "loss": 1.521,
381
+ "step": 265
382
+ },
383
+ {
384
+ "epoch": 0.5339925834363412,
385
+ "grad_norm": 2.4375,
386
+ "learning_rate": 0.0002,
387
+ "loss": 1.4559,
388
+ "step": 270
389
+ },
390
+ {
391
+ "epoch": 0.5438813349814586,
392
+ "grad_norm": 2.5,
393
+ "learning_rate": 0.0002,
394
+ "loss": 1.4516,
395
+ "step": 275
396
+ },
397
+ {
398
+ "epoch": 0.553770086526576,
399
+ "grad_norm": 2.125,
400
+ "learning_rate": 0.0002,
401
+ "loss": 1.3968,
402
+ "step": 280
403
+ },
404
+ {
405
+ "epoch": 0.5636588380716935,
406
+ "grad_norm": 2.328125,
407
+ "learning_rate": 0.0002,
408
+ "loss": 1.4183,
409
+ "step": 285
410
+ },
411
+ {
412
+ "epoch": 0.5735475896168108,
413
+ "grad_norm": 1.8046875,
414
+ "learning_rate": 0.0002,
415
+ "loss": 1.3653,
416
+ "step": 290
417
+ },
418
+ {
419
+ "epoch": 0.5834363411619283,
420
+ "grad_norm": 2.0,
421
+ "learning_rate": 0.0002,
422
+ "loss": 1.388,
423
+ "step": 295
424
+ },
425
+ {
426
+ "epoch": 0.5933250927070457,
427
+ "grad_norm": 1.953125,
428
+ "learning_rate": 0.0002,
429
+ "loss": 1.3131,
430
+ "step": 300
431
+ },
432
+ {
433
+ "epoch": 0.6032138442521632,
434
+ "grad_norm": 2.390625,
435
+ "learning_rate": 0.0002,
436
+ "loss": 1.4784,
437
+ "step": 305
438
+ },
439
+ {
440
+ "epoch": 0.6131025957972805,
441
+ "grad_norm": 2.34375,
442
+ "learning_rate": 0.0002,
443
+ "loss": 1.4339,
444
+ "step": 310
445
+ },
446
+ {
447
+ "epoch": 0.622991347342398,
448
+ "grad_norm": 2.140625,
449
+ "learning_rate": 0.0002,
450
+ "loss": 1.4425,
451
+ "step": 315
452
+ },
453
+ {
454
+ "epoch": 0.6328800988875154,
455
+ "grad_norm": 2.078125,
456
+ "learning_rate": 0.0002,
457
+ "loss": 1.3847,
458
+ "step": 320
459
+ },
460
+ {
461
+ "epoch": 0.6427688504326329,
462
+ "grad_norm": 2.359375,
463
+ "learning_rate": 0.0002,
464
+ "loss": 1.4252,
465
+ "step": 325
466
+ },
467
+ {
468
+ "epoch": 0.6526576019777504,
469
+ "grad_norm": 2.640625,
470
+ "learning_rate": 0.0002,
471
+ "loss": 1.4253,
472
+ "step": 330
473
+ },
474
+ {
475
+ "epoch": 0.6625463535228677,
476
+ "grad_norm": 2.015625,
477
+ "learning_rate": 0.0002,
478
+ "loss": 1.3336,
479
+ "step": 335
480
+ },
481
+ {
482
+ "epoch": 0.6724351050679852,
483
+ "grad_norm": 2.171875,
484
+ "learning_rate": 0.0002,
485
+ "loss": 1.4209,
486
+ "step": 340
487
+ },
488
+ {
489
+ "epoch": 0.6823238566131026,
490
+ "grad_norm": 4.0625,
491
+ "learning_rate": 0.0002,
492
+ "loss": 1.4485,
493
+ "step": 345
494
+ },
495
+ {
496
+ "epoch": 0.69221260815822,
497
+ "grad_norm": 2.1875,
498
+ "learning_rate": 0.0002,
499
+ "loss": 1.4261,
500
+ "step": 350
501
+ },
502
+ {
503
+ "epoch": 0.7021013597033374,
504
+ "grad_norm": 2.515625,
505
+ "learning_rate": 0.0002,
506
+ "loss": 1.3869,
507
+ "step": 355
508
+ },
509
+ {
510
+ "epoch": 0.7119901112484549,
511
+ "grad_norm": 2.1875,
512
+ "learning_rate": 0.0002,
513
+ "loss": 1.4681,
514
+ "step": 360
515
+ },
516
+ {
517
+ "epoch": 0.7218788627935723,
518
+ "grad_norm": 3.609375,
519
+ "learning_rate": 0.0002,
520
+ "loss": 1.3513,
521
+ "step": 365
522
+ },
523
+ {
524
+ "epoch": 0.7317676143386898,
525
+ "grad_norm": 2.296875,
526
+ "learning_rate": 0.0002,
527
+ "loss": 1.4029,
528
+ "step": 370
529
+ },
530
+ {
531
+ "epoch": 0.7416563658838071,
532
+ "grad_norm": 2.15625,
533
+ "learning_rate": 0.0002,
534
+ "loss": 1.3716,
535
+ "step": 375
536
+ },
537
+ {
538
+ "epoch": 0.7515451174289246,
539
+ "grad_norm": 3.75,
540
+ "learning_rate": 0.0002,
541
+ "loss": 1.3902,
542
+ "step": 380
543
+ },
544
+ {
545
+ "epoch": 0.761433868974042,
546
+ "grad_norm": 2.0625,
547
+ "learning_rate": 0.0002,
548
+ "loss": 1.3469,
549
+ "step": 385
550
+ },
551
+ {
552
+ "epoch": 0.7713226205191595,
553
+ "grad_norm": 1.9609375,
554
+ "learning_rate": 0.0002,
555
+ "loss": 1.3216,
556
+ "step": 390
557
+ },
558
+ {
559
+ "epoch": 0.7812113720642769,
560
+ "grad_norm": 1.6796875,
561
+ "learning_rate": 0.0002,
562
+ "loss": 1.4774,
563
+ "step": 395
564
+ },
565
+ {
566
+ "epoch": 0.7911001236093943,
567
+ "grad_norm": 1.7265625,
568
+ "learning_rate": 0.0002,
569
+ "loss": 1.372,
570
+ "step": 400
571
+ },
572
+ {
573
+ "epoch": 0.8009888751545118,
574
+ "grad_norm": 1.921875,
575
+ "learning_rate": 0.0002,
576
+ "loss": 1.3574,
577
+ "step": 405
578
+ },
579
+ {
580
+ "epoch": 0.8108776266996292,
581
+ "grad_norm": 2.1875,
582
+ "learning_rate": 0.0002,
583
+ "loss": 1.4444,
584
+ "step": 410
585
+ },
586
+ {
587
+ "epoch": 0.8207663782447466,
588
+ "grad_norm": 2.296875,
589
+ "learning_rate": 0.0002,
590
+ "loss": 1.3743,
591
+ "step": 415
592
+ },
593
+ {
594
+ "epoch": 0.830655129789864,
595
+ "grad_norm": 2.125,
596
+ "learning_rate": 0.0002,
597
+ "loss": 1.3589,
598
+ "step": 420
599
+ },
600
+ {
601
+ "epoch": 0.8405438813349815,
602
+ "grad_norm": 2.046875,
603
+ "learning_rate": 0.0002,
604
+ "loss": 1.3336,
605
+ "step": 425
606
+ },
607
+ {
608
+ "epoch": 0.8504326328800988,
609
+ "grad_norm": 3.84375,
610
+ "learning_rate": 0.0002,
611
+ "loss": 1.3584,
612
+ "step": 430
613
+ },
614
+ {
615
+ "epoch": 0.8603213844252163,
616
+ "grad_norm": 2.125,
617
+ "learning_rate": 0.0002,
618
+ "loss": 1.332,
619
+ "step": 435
620
+ },
621
+ {
622
+ "epoch": 0.8702101359703337,
623
+ "grad_norm": 1.8828125,
624
+ "learning_rate": 0.0002,
625
+ "loss": 1.3678,
626
+ "step": 440
627
+ },
628
+ {
629
+ "epoch": 0.8800988875154512,
630
+ "grad_norm": 2.0625,
631
+ "learning_rate": 0.0002,
632
+ "loss": 1.2967,
633
+ "step": 445
634
+ },
635
+ {
636
+ "epoch": 0.8899876390605687,
637
+ "grad_norm": 2.3125,
638
+ "learning_rate": 0.0002,
639
+ "loss": 1.3276,
640
+ "step": 450
641
+ },
642
+ {
643
+ "epoch": 0.899876390605686,
644
+ "grad_norm": 1.953125,
645
+ "learning_rate": 0.0002,
646
+ "loss": 1.3175,
647
+ "step": 455
648
+ },
649
+ {
650
+ "epoch": 0.9097651421508035,
651
+ "grad_norm": 1.7734375,
652
+ "learning_rate": 0.0002,
653
+ "loss": 1.366,
654
+ "step": 460
655
+ },
656
+ {
657
+ "epoch": 0.9196538936959209,
658
+ "grad_norm": 2.46875,
659
+ "learning_rate": 0.0002,
660
+ "loss": 1.3105,
661
+ "step": 465
662
+ },
663
+ {
664
+ "epoch": 0.9295426452410384,
665
+ "grad_norm": 1.84375,
666
+ "learning_rate": 0.0002,
667
+ "loss": 1.2856,
668
+ "step": 470
669
+ },
670
+ {
671
+ "epoch": 0.9394313967861557,
672
+ "grad_norm": 1.7890625,
673
+ "learning_rate": 0.0002,
674
+ "loss": 1.3662,
675
+ "step": 475
676
+ },
677
+ {
678
+ "epoch": 0.9493201483312732,
679
+ "grad_norm": 1.921875,
680
+ "learning_rate": 0.0002,
681
+ "loss": 1.3289,
682
+ "step": 480
683
+ },
684
+ {
685
+ "epoch": 0.9592088998763906,
686
+ "grad_norm": 2.078125,
687
+ "learning_rate": 0.0002,
688
+ "loss": 1.3812,
689
+ "step": 485
690
+ },
691
+ {
692
+ "epoch": 0.969097651421508,
693
+ "grad_norm": 2.703125,
694
+ "learning_rate": 0.0002,
695
+ "loss": 1.3344,
696
+ "step": 490
697
+ },
698
+ {
699
+ "epoch": 0.9789864029666254,
700
+ "grad_norm": 1.9609375,
701
+ "learning_rate": 0.0002,
702
+ "loss": 1.345,
703
+ "step": 495
704
+ },
705
+ {
706
+ "epoch": 0.9888751545117429,
707
+ "grad_norm": 2.125,
708
+ "learning_rate": 0.0002,
709
+ "loss": 1.3163,
710
+ "step": 500
711
+ },
712
+ {
713
+ "epoch": 0.9987639060568603,
714
+ "grad_norm": 2.125,
715
+ "learning_rate": 0.0002,
716
+ "loss": 1.354,
717
+ "step": 505
718
+ },
719
+ {
720
+ "epoch": 0.9987639060568603,
721
+ "step": 505,
722
+ "total_flos": 9435624701952000.0,
723
+ "train_loss": 1.491983689412032,
724
+ "train_runtime": 449.9567,
725
+ "train_samples_per_second": 17.977,
726
+ "train_steps_per_second": 1.122
727
+ }
728
+ ],
729
+ "logging_steps": 5,
730
+ "max_steps": 505,
731
+ "num_input_tokens_seen": 0,
732
+ "num_train_epochs": 1,
733
+ "save_steps": 100,
734
+ "stateful_callbacks": {
735
+ "TrainerControl": {
736
+ "args": {
737
+ "should_epoch_stop": false,
738
+ "should_evaluate": false,
739
+ "should_log": false,
740
+ "should_save": true,
741
+ "should_training_stop": true
742
+ },
743
+ "attributes": {}
744
+ }
745
+ },
746
+ "total_flos": 9435624701952000.0,
747
+ "train_batch_size": 2,
748
+ "trial_name": null,
749
+ "trial_params": null
750
+ }