Hugging Face's logo

obiwit
/
llama3.2-3b-sft-full

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
trl
sft
conversational
text-generation-inference
Model card Files
xet
 Community
obiwit commited on
Commit
3e08612
·
verified ·
1 Parent(s): 4360757

Model save

Browse files
Files changed (7) hide show
  1. README.md +57 -0
  2. all_results.json +9 -0
  3. generation_config.json +9 -0
  4. model-00001-of-00002.safetensors +1 -1
  5. model-00002-of-00002.safetensors +1 -1
  6. train_results.json +9 -0
  7. trainer_state.json +787 -0
README.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: meta-llama/Llama-3.2-3B
3
+ library_name: transformers
4
+ model_name: llama3.2-3b-sft-full
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - sft
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for llama3.2-3b-sft-full
13
+
14
+ This model is a fine-tuned version of [meta-llama/Llama-3.2-3B](https://huggingface.co/meta-llama/Llama-3.2-3B).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="obiwit/llama3.2-3b-sft-full", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/bborges/L3-8B_preferences/runs/pnv8ca52)
31
+
32
+ This model was trained with SFT.
33
+
34
+ ### Framework versions
35
+
36
+ - TRL: 0.12.2
37
+ - Transformers: 4.46.3
38
+ - Pytorch: 2.1.2+cu121
39
+ - Datasets: 3.1.0
40
+ - Tokenizers: 0.20.3
41
+
42
+ ## Citations
43
+
44
+
45
+
46
+ Cite TRL as:
47
+
48
+ ```bibtex
49
+ @misc{vonwerra2022trl,
50
+ title = {{TRL: Transformer Reinforcement Learning}},
51
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
52
+ year = 2020,
53
+ journal = {GitHub repository},
54
+ publisher = {GitHub},
55
+ howpublished = {\url{https://github.com/huggingface/trl}}
56
+ }
57
+ ```
all_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 10.0,
3
+ "total_flos": 651355949629440.0,
4
+ "train_loss": 0.5446855222952038,
5
+ "train_runtime": 13348.3684,
6
+ "train_samples": 164653,
7
+ "train_samples_per_second": 45.308,
8
+ "train_steps_per_second": 0.354
9
+ }
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 128000,
4
+ "do_sample": true,
5
+ "eos_token_id": 128001,
6
+ "temperature": 0.6,
7
+ "top_p": 0.9,
8
+ "transformers_version": "4.46.3"
9
+ }
model-00001-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bc37e2c0c88c7ab1e7d84e513aabaf1179db7673211cef2d4b19ecb0280fd399
3
  size 4965799096
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:33192c33081e847991209ca36277a2f010e5cb1f7a30aad3eb728000f95310c0
3
  size 4965799096
model-00002-of-00002.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:cf18ccbefc1b0641309740913aa3e69349ad7392071acf0999db4afaf4c96f55
3
  size 1459729952
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b42807ccde702bb0aa3f1c056176b4950e29415d2c26be01fc356288c0794c09
3
  size 1459729952
train_results.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "epoch": 10.0,
3
+ "total_flos": 651355949629440.0,
4
+ "train_loss": 0.5446855222952038,
5
+ "train_runtime": 13348.3684,
6
+ "train_samples": 164653,
7
+ "train_samples_per_second": 45.308,
8
+ "train_steps_per_second": 0.354
9
+ }
trainer_state.json ADDED
@@ -0,0 +1,787 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": 1.2029809951782227,
3
+ "best_model_checkpoint": "models/llama3.2-3b-sft-full/checkpoint-946",
4
+ "epoch": 10.0,
5
+ "eval_steps": 500,
6
+ "global_step": 4730,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 0.0021141649048625794,
13
+ "grad_norm": 12.750687916050717,
14
+ "learning_rate": 4.228329809725159e-08,
15
+ "loss": 2.0313,
16
+ "step": 1
17
+ },
18
+ {
19
+ "epoch": 0.10570824524312897,
20
+ "grad_norm": 1.6188098557526023,
21
+ "learning_rate": 2.1141649048625796e-06,
22
+ "loss": 1.8564,
23
+ "step": 50
24
+ },
25
+ {
26
+ "epoch": 0.21141649048625794,
27
+ "grad_norm": 1.3518202018434258,
28
+ "learning_rate": 4.228329809725159e-06,
29
+ "loss": 1.5595,
30
+ "step": 100
31
+ },
32
+ {
33
+ "epoch": 0.3171247357293869,
34
+ "grad_norm": 1.0657590959696566,
35
+ "learning_rate": 6.342494714587738e-06,
36
+ "loss": 1.4551,
37
+ "step": 150
38
+ },
39
+ {
40
+ "epoch": 0.42283298097251587,
41
+ "grad_norm": 0.8592382874171366,
42
+ "learning_rate": 8.456659619450318e-06,
43
+ "loss": 1.3854,
44
+ "step": 200
45
+ },
46
+ {
47
+ "epoch": 0.5285412262156448,
48
+ "grad_norm": 0.9009389463159452,
49
+ "learning_rate": 1.0570824524312897e-05,
50
+ "loss": 1.3425,
51
+ "step": 250
52
+ },
53
+ {
54
+ "epoch": 0.6342494714587738,
55
+ "grad_norm": 1.1412749380165943,
56
+ "learning_rate": 1.2684989429175477e-05,
57
+ "loss": 1.3006,
58
+ "step": 300
59
+ },
60
+ {
61
+ "epoch": 0.7399577167019028,
62
+ "grad_norm": 0.7983643996119608,
63
+ "learning_rate": 1.4799154334038057e-05,
64
+ "loss": 1.2744,
65
+ "step": 350
66
+ },
67
+ {
68
+ "epoch": 0.8456659619450317,
69
+ "grad_norm": 0.8244042644021615,
70
+ "learning_rate": 1.6913319238900637e-05,
71
+ "loss": 1.2548,
72
+ "step": 400
73
+ },
74
+ {
75
+ "epoch": 0.9513742071881607,
76
+ "grad_norm": 0.8216465876347789,
77
+ "learning_rate": 1.9027484143763216e-05,
78
+ "loss": 1.2404,
79
+ "step": 450
80
+ },
81
+ {
82
+ "epoch": 1.0,
83
+ "eval_loss": 1.2400952577590942,
84
+ "eval_runtime": 18.7515,
85
+ "eval_samples_per_second": 180.785,
86
+ "eval_steps_per_second": 1.44,
87
+ "step": 473
88
+ },
89
+ {
90
+ "epoch": 1.0570824524312896,
91
+ "grad_norm": 0.8252930310615743,
92
+ "learning_rate": 1.9998014930992976e-05,
93
+ "loss": 1.1705,
94
+ "step": 500
95
+ },
96
+ {
97
+ "epoch": 1.1627906976744187,
98
+ "grad_norm": 0.8757450644495188,
99
+ "learning_rate": 1.9983859126509827e-05,
100
+ "loss": 1.12,
101
+ "step": 550
102
+ },
103
+ {
104
+ "epoch": 1.2684989429175475,
105
+ "grad_norm": 0.7949433367488117,
106
+ "learning_rate": 1.9956111376617147e-05,
107
+ "loss": 1.1182,
108
+ "step": 600
109
+ },
110
+ {
111
+ "epoch": 1.3742071881606766,
112
+ "grad_norm": 0.8258557786713708,
113
+ "learning_rate": 1.9914809456878166e-05,
114
+ "loss": 1.1182,
115
+ "step": 650
116
+ },
117
+ {
118
+ "epoch": 1.4799154334038054,
119
+ "grad_norm": 0.7933398639276036,
120
+ "learning_rate": 1.9860009595393798e-05,
121
+ "loss": 1.1038,
122
+ "step": 700
123
+ },
124
+ {
125
+ "epoch": 1.5856236786469344,
126
+ "grad_norm": 0.841899074998612,
127
+ "learning_rate": 1.9791786396254126e-05,
128
+ "loss": 1.1021,
129
+ "step": 750
130
+ },
131
+ {
132
+ "epoch": 1.6913319238900635,
133
+ "grad_norm": 0.7846336500672028,
134
+ "learning_rate": 1.971023273797303e-05,
135
+ "loss": 1.1144,
136
+ "step": 800
137
+ },
138
+ {
139
+ "epoch": 1.7970401691331923,
140
+ "grad_norm": 0.8229424691925108,
141
+ "learning_rate": 1.961545964704409e-05,
142
+ "loss": 1.1026,
143
+ "step": 850
144
+ },
145
+ {
146
+ "epoch": 1.9027484143763214,
147
+ "grad_norm": 0.7508672417042123,
148
+ "learning_rate": 1.950759614679005e-05,
149
+ "loss": 1.0999,
150
+ "step": 900
151
+ },
152
+ {
153
+ "epoch": 2.0,
154
+ "eval_loss": 1.2029809951782227,
155
+ "eval_runtime": 18.6697,
156
+ "eval_samples_per_second": 181.578,
157
+ "eval_steps_per_second": 1.446,
158
+ "step": 946
159
+ },
160
+ {
161
+ "epoch": 2.0084566596194504,
162
+ "grad_norm": 1.1731388344496063,
163
+ "learning_rate": 1.9386789081711465e-05,
164
+ "loss": 1.0829,
165
+ "step": 950
166
+ },
167
+ {
168
+ "epoch": 2.1141649048625792,
169
+ "grad_norm": 0.8445527906517972,
170
+ "learning_rate": 1.9253202917573813e-05,
171
+ "loss": 0.8792,
172
+ "step": 1000
173
+ },
174
+ {
175
+ "epoch": 2.219873150105708,
176
+ "grad_norm": 0.8760066547012747,
177
+ "learning_rate": 1.910701951750511e-05,
178
+ "loss": 0.8858,
179
+ "step": 1050
180
+ },
181
+ {
182
+ "epoch": 2.3255813953488373,
183
+ "grad_norm": 0.8556944230458313,
184
+ "learning_rate": 1.894843789440892e-05,
185
+ "loss": 0.8963,
186
+ "step": 1100
187
+ },
188
+ {
189
+ "epoch": 2.431289640591966,
190
+ "grad_norm": 0.9640553709381205,
191
+ "learning_rate": 1.8777673940029783e-05,
192
+ "loss": 0.8986,
193
+ "step": 1150
194
+ },
195
+ {
196
+ "epoch": 2.536997885835095,
197
+ "grad_norm": 0.8627140770648074,
198
+ "learning_rate": 1.8594960131039935e-05,
199
+ "loss": 0.8938,
200
+ "step": 1200
201
+ },
202
+ {
203
+ "epoch": 2.6427061310782243,
204
+ "grad_norm": 0.9535715556479282,
205
+ "learning_rate": 1.8400545212547452e-05,
206
+ "loss": 0.8955,
207
+ "step": 1250
208
+ },
209
+ {
210
+ "epoch": 2.748414376321353,
211
+ "grad_norm": 0.8552691365971952,
212
+ "learning_rate": 1.819469385945664e-05,
213
+ "loss": 0.9014,
214
+ "step": 1300
215
+ },
216
+ {
217
+ "epoch": 2.854122621564482,
218
+ "grad_norm": 0.8665945027227095,
219
+ "learning_rate": 1.7977686316141757e-05,
220
+ "loss": 0.8947,
221
+ "step": 1350
222
+ },
223
+ {
224
+ "epoch": 2.9598308668076108,
225
+ "grad_norm": 0.9619834334611845,
226
+ "learning_rate": 1.7749818014924612e-05,
227
+ "loss": 0.9009,
228
+ "step": 1400
229
+ },
230
+ {
231
+ "epoch": 3.0,
232
+ "eval_loss": 1.2264167070388794,
233
+ "eval_runtime": 18.611,
234
+ "eval_samples_per_second": 182.15,
235
+ "eval_steps_per_second": 1.451,
236
+ "step": 1419
237
+ },
238
+ {
239
+ "epoch": 3.06553911205074,
240
+ "grad_norm": 1.0859362155871017,
241
+ "learning_rate": 1.7511399173875326e-05,
242
+ "loss": 0.7572,
243
+ "step": 1450
244
+ },
245
+ {
246
+ "epoch": 3.171247357293869,
247
+ "grad_norm": 4.277266247206096,
248
+ "learning_rate": 1.7262754374483997e-05,
249
+ "loss": 0.667,
250
+ "step": 1500
251
+ },
252
+ {
253
+ "epoch": 3.276955602536998,
254
+ "grad_norm": 1.0657798474808275,
255
+ "learning_rate": 1.7004222119778044e-05,
256
+ "loss": 0.6762,
257
+ "step": 1550
258
+ },
259
+ {
260
+ "epoch": 3.382663847780127,
261
+ "grad_norm": 1.1532443588160448,
262
+ "learning_rate": 1.673615437348693e-05,
263
+ "loss": 0.6818,
264
+ "step": 1600
265
+ },
266
+ {
267
+ "epoch": 3.488372093023256,
268
+ "grad_norm": 1.027155281075861,
269
+ "learning_rate": 1.6458916080881566e-05,
270
+ "loss": 0.6855,
271
+ "step": 1650
272
+ },
273
+ {
274
+ "epoch": 3.5940803382663846,
275
+ "grad_norm": 1.1083716056629094,
276
+ "learning_rate": 1.6172884671940753e-05,
277
+ "loss": 0.6812,
278
+ "step": 1700
279
+ },
280
+ {
281
+ "epoch": 3.699788583509514,
282
+ "grad_norm": 1.0062825632251786,
283
+ "learning_rate": 1.587844954752106e-05,
284
+ "loss": 0.6883,
285
+ "step": 1750
286
+ },
287
+ {
288
+ "epoch": 3.8054968287526427,
289
+ "grad_norm": 1.07000970987697,
290
+ "learning_rate": 1.557601154922964e-05,
291
+ "loss": 0.6893,
292
+ "step": 1800
293
+ },
294
+ {
295
+ "epoch": 3.9112050739957716,
296
+ "grad_norm": 1.0384275987977813,
297
+ "learning_rate": 1.5265982413721662e-05,
298
+ "loss": 0.6876,
299
+ "step": 1850
300
+ },
301
+ {
302
+ "epoch": 4.0,
303
+ "eval_loss": 1.2967802286148071,
304
+ "eval_runtime": 18.6163,
305
+ "eval_samples_per_second": 182.098,
306
+ "eval_steps_per_second": 1.45,
307
+ "step": 1892
308
+ },
309
+ {
310
+ "epoch": 4.016913319238901,
311
+ "grad_norm": 1.3109377397084605,
312
+ "learning_rate": 1.494878421216539e-05,
313
+ "loss": 0.6596,
314
+ "step": 1900
315
+ },
316
+ {
317
+ "epoch": 4.12262156448203,
318
+ "grad_norm": 1.2688929301953065,
319
+ "learning_rate": 1.4624848775637845e-05,
320
+ "loss": 0.4741,
321
+ "step": 1950
322
+ },
323
+ {
324
+ "epoch": 4.2283298097251585,
325
+ "grad_norm": 1.1894927227873353,
326
+ "learning_rate": 1.4294617107233504e-05,
327
+ "loss": 0.4744,
328
+ "step": 2000
329
+ },
330
+ {
331
+ "epoch": 4.334038054968287,
332
+ "grad_norm": 1.2032936700401937,
333
+ "learning_rate": 1.3958538781686194e-05,
334
+ "loss": 0.4805,
335
+ "step": 2050
336
+ },
337
+ {
338
+ "epoch": 4.439746300211416,
339
+ "grad_norm": 1.3185879928970223,
340
+ "learning_rate": 1.3617071333321705e-05,
341
+ "loss": 0.4826,
342
+ "step": 2100
343
+ },
344
+ {
345
+ "epoch": 4.545454545454545,
346
+ "grad_norm": 1.209922790479991,
347
+ "learning_rate": 1.3270679633174219e-05,
348
+ "loss": 0.4839,
349
+ "step": 2150
350
+ },
351
+ {
352
+ "epoch": 4.651162790697675,
353
+ "grad_norm": 1.242791543340292,
354
+ "learning_rate": 1.2919835256114639e-05,
355
+ "loss": 0.4885,
356
+ "step": 2200
357
+ },
358
+ {
359
+ "epoch": 4.7568710359408035,
360
+ "grad_norm": 1.201986398108636,
361
+ "learning_rate": 1.2565015838852364e-05,
362
+ "loss": 0.4871,
363
+ "step": 2250
364
+ },
365
+ {
366
+ "epoch": 4.862579281183932,
367
+ "grad_norm": 1.2360526015797055,
368
+ "learning_rate": 1.2206704429684504e-05,
369
+ "loss": 0.4846,
370
+ "step": 2300
371
+ },
372
+ {
373
+ "epoch": 4.968287526427061,
374
+ "grad_norm": 1.1786633423072006,
375
+ "learning_rate": 1.1845388830877826e-05,
376
+ "loss": 0.4893,
377
+ "step": 2350
378
+ },
379
+ {
380
+ "epoch": 5.0,
381
+ "eval_loss": 1.4134972095489502,
382
+ "eval_runtime": 18.68,
383
+ "eval_samples_per_second": 181.478,
384
+ "eval_steps_per_second": 1.445,
385
+ "step": 2365
386
+ },
387
+ {
388
+ "epoch": 5.07399577167019,
389
+ "grad_norm": 1.2309864025937378,
390
+ "learning_rate": 1.1481560934578686e-05,
391
+ "loss": 0.3702,
392
+ "step": 2400
393
+ },
394
+ {
395
+ "epoch": 5.179704016913319,
396
+ "grad_norm": 1.2786650363864003,
397
+ "learning_rate": 1.1115716053155003e-05,
398
+ "loss": 0.3218,
399
+ "step": 2450
400
+ },
401
+ {
402
+ "epoch": 5.2854122621564485,
403
+ "grad_norm": 1.2330334913778382,
404
+ "learning_rate": 1.0748352244882008e-05,
405
+ "loss": 0.3265,
406
+ "step": 2500
407
+ },
408
+ {
409
+ "epoch": 5.391120507399577,
410
+ "grad_norm": 1.2643477957547729,
411
+ "learning_rate": 1.0379969635889705e-05,
412
+ "loss": 0.3295,
413
+ "step": 2550
414
+ },
415
+ {
416
+ "epoch": 5.496828752642706,
417
+ "grad_norm": 1.281027075874609,
418
+ "learning_rate": 1.0011069739295196e-05,
419
+ "loss": 0.3263,
420
+ "step": 2600
421
+ },
422
+ {
423
+ "epoch": 5.602536997885835,
424
+ "grad_norm": 1.22994140794609,
425
+ "learning_rate": 9.642154772446763e-06,
426
+ "loss": 0.3289,
427
+ "step": 2650
428
+ },
429
+ {
430
+ "epoch": 5.708245243128964,
431
+ "grad_norm": 1.3028039578218702,
432
+ "learning_rate": 9.273726973209203e-06,
433
+ "loss": 0.3284,
434
+ "step": 2700
435
+ },
436
+ {
437
+ "epoch": 5.813953488372093,
438
+ "grad_norm": 1.2532248232758287,
439
+ "learning_rate": 8.906287916221259e-06,
440
+ "loss": 0.3308,
441
+ "step": 2750
442
+ },
443
+ {
444
+ "epoch": 5.9196617336152215,
445
+ "grad_norm": 1.3100012552937503,
446
+ "learning_rate": 8.540337830055927e-06,
447
+ "loss": 0.3315,
448
+ "step": 2800
449
+ },
450
+ {
451
+ "epoch": 6.0,
452
+ "eval_loss": 1.5860239267349243,
453
+ "eval_runtime": 18.5094,
454
+ "eval_samples_per_second": 183.151,
455
+ "eval_steps_per_second": 1.459,
456
+ "step": 2838
457
+ },
458
+ {
459
+ "epoch": 6.025369978858351,
460
+ "grad_norm": 1.4354000732242087,
461
+ "learning_rate": 8.176374916213325e-06,
462
+ "loss": 0.3032,
463
+ "step": 2850
464
+ },
465
+ {
466
+ "epoch": 6.13107822410148,
467
+ "grad_norm": 1.1648180451771042,
468
+ "learning_rate": 7.814894670873171e-06,
469
+ "loss": 0.2141,
470
+ "step": 2900
471
+ },
472
+ {
473
+ "epoch": 6.236786469344609,
474
+ "grad_norm": 1.1884819713548493,
475
+ "learning_rate": 7.456389210330289e-06,
476
+ "loss": 0.2153,
477
+ "step": 2950
478
+ },
479
+ {
480
+ "epoch": 6.342494714587738,
481
+ "grad_norm": 1.2643468298433007,
482
+ "learning_rate": 7.101346601031416e-06,
483
+ "loss": 0.2153,
484
+ "step": 3000
485
+ },
486
+ {
487
+ "epoch": 6.4482029598308666,
488
+ "grad_norm": 1.2064695562383054,
489
+ "learning_rate": 6.750250195125509e-06,
490
+ "loss": 0.2188,
491
+ "step": 3050
492
+ },
493
+ {
494
+ "epoch": 6.553911205073996,
495
+ "grad_norm": 1.2016268195975075,
496
+ "learning_rate": 6.4035779724320115e-06,
497
+ "loss": 0.2171,
498
+ "step": 3100
499
+ },
500
+ {
501
+ "epoch": 6.659619450317125,
502
+ "grad_norm": 1.2411099570397714,
503
+ "learning_rate": 6.061801889722952e-06,
504
+ "loss": 0.2162,
505
+ "step": 3150
506
+ },
507
+ {
508
+ "epoch": 6.765327695560254,
509
+ "grad_norm": 1.2392777259407999,
510
+ "learning_rate": 5.725387238204831e-06,
511
+ "loss": 0.215,
512
+ "step": 3200
513
+ },
514
+ {
515
+ "epoch": 6.871035940803383,
516
+ "grad_norm": 1.1740909434133877,
517
+ "learning_rate": 5.394792010074907e-06,
518
+ "loss": 0.2166,
519
+ "step": 3250
520
+ },
521
+ {
522
+ "epoch": 6.976744186046512,
523
+ "grad_norm": 1.2762193770401755,
524
+ "learning_rate": 5.0704662750142875e-06,
525
+ "loss": 0.2165,
526
+ "step": 3300
527
+ },
528
+ {
529
+ "epoch": 7.0,
530
+ "eval_loss": 1.802822470664978,
531
+ "eval_runtime": 18.5091,
532
+ "eval_samples_per_second": 183.153,
533
+ "eval_steps_per_second": 1.459,
534
+ "step": 3311
535
+ },
536
+ {
537
+ "epoch": 7.08245243128964,
538
+ "grad_norm": 1.0526386037599447,
539
+ "learning_rate": 4.752851567466723e-06,
540
+ "loss": 0.1594,
541
+ "step": 3350
542
+ },
543
+ {
544
+ "epoch": 7.188160676532769,
545
+ "grad_norm": 1.147178748636373,
546
+ "learning_rate": 4.442380285537152e-06,
547
+ "loss": 0.1424,
548
+ "step": 3400
549
+ },
550
+ {
551
+ "epoch": 7.293868921775898,
552
+ "grad_norm": 1.0933846169501016,
553
+ "learning_rate": 4.1394751023283985e-06,
554
+ "loss": 0.1429,
555
+ "step": 3450
556
+ },
557
+ {
558
+ "epoch": 7.399577167019028,
559
+ "grad_norm": 1.1027491437777261,
560
+ "learning_rate": 3.844548390517433e-06,
561
+ "loss": 0.1446,
562
+ "step": 3500
563
+ },
564
+ {
565
+ "epoch": 7.505285412262157,
566
+ "grad_norm": 1.17228307610312,
567
+ "learning_rate": 3.5580016609545076e-06,
568
+ "loss": 0.1472,
569
+ "step": 3550
570
+ },
571
+ {
572
+ "epoch": 7.6109936575052854,
573
+ "grad_norm": 1.1469564201071871,
574
+ "learning_rate": 3.280225016049543e-06,
575
+ "loss": 0.1444,
576
+ "step": 3600
577
+ },
578
+ {
579
+ "epoch": 7.716701902748414,
580
+ "grad_norm": 1.1481692114896938,
581
+ "learning_rate": 3.011596618689825e-06,
582
+ "loss": 0.1442,
583
+ "step": 3650
584
+ },
585
+ {
586
+ "epoch": 7.822410147991543,
587
+ "grad_norm": 1.0576784331195976,
588
+ "learning_rate": 2.7524821774121158e-06,
589
+ "loss": 0.1432,
590
+ "step": 3700
591
+ },
592
+ {
593
+ "epoch": 7.928118393234672,
594
+ "grad_norm": 1.017768412790375,
595
+ "learning_rate": 2.5032344485299886e-06,
596
+ "loss": 0.1444,
597
+ "step": 3750
598
+ },
599
+ {
600
+ "epoch": 8.0,
601
+ "eval_loss": 2.0527853965759277,
602
+ "eval_runtime": 18.5305,
603
+ "eval_samples_per_second": 182.942,
604
+ "eval_steps_per_second": 1.457,
605
+ "step": 3784
606
+ },
607
+ {
608
+ "epoch": 8.033826638477802,
609
+ "grad_norm": 0.8182949486515412,
610
+ "learning_rate": 2.2641927558942135e-06,
611
+ "loss": 0.1304,
612
+ "step": 3800
613
+ },
614
+ {
615
+ "epoch": 8.13953488372093,
616
+ "grad_norm": 0.8622437048349989,
617
+ "learning_rate": 2.0356825289400185e-06,
618
+ "loss": 0.1036,
619
+ "step": 3850
620
+ },
621
+ {
622
+ "epoch": 8.24524312896406,
623
+ "grad_norm": 0.7861265978157033,
624
+ "learning_rate": 1.818014859650068e-06,
625
+ "loss": 0.1028,
626
+ "step": 3900
627
+ },
628
+ {
629
+ "epoch": 8.350951374207188,
630
+ "grad_norm": 0.8194211537002656,
631
+ "learning_rate": 1.6114860790363351e-06,
632
+ "loss": 0.1032,
633
+ "step": 3950
634
+ },
635
+ {
636
+ "epoch": 8.456659619450317,
637
+ "grad_norm": 0.8182271224247505,
638
+ "learning_rate": 1.4163773537174653e-06,
639
+ "loss": 0.1032,
640
+ "step": 4000
641
+ },
642
+ {
643
+ "epoch": 8.562367864693446,
644
+ "grad_norm": 0.8307023639288861,
645
+ "learning_rate": 1.2329543031407953e-06,
646
+ "loss": 0.1036,
647
+ "step": 4050
648
+ },
649
+ {
650
+ "epoch": 8.668076109936575,
651
+ "grad_norm": 0.8784936190909071,
652
+ "learning_rate": 1.0614666379701732e-06,
653
+ "loss": 0.1023,
654
+ "step": 4100
655
+ },
656
+ {
657
+ "epoch": 8.773784355179703,
658
+ "grad_norm": 0.81990506328911,
659
+ "learning_rate": 9.021478201318846e-07,
660
+ "loss": 0.1036,
661
+ "step": 4150
662
+ },
663
+ {
664
+ "epoch": 8.879492600422832,
665
+ "grad_norm": 1.2592556041877936,
666
+ "learning_rate": 7.552147449814573e-07,
667
+ "loss": 0.1044,
668
+ "step": 4200
669
+ },
670
+ {
671
+ "epoch": 8.985200845665961,
672
+ "grad_norm": 0.7975519834622627,
673
+ "learning_rate": 6.20867446024066e-07,
674
+ "loss": 0.1032,
675
+ "step": 4250
676
+ },
677
+ {
678
+ "epoch": 9.0,
679
+ "eval_loss": 2.279863119125366,
680
+ "eval_runtime": 18.5289,
681
+ "eval_samples_per_second": 182.957,
682
+ "eval_steps_per_second": 1.457,
683
+ "step": 4257
684
+ },
685
+ {
686
+ "epoch": 9.090909090909092,
687
+ "grad_norm": 0.7603748072458637,
688
+ "learning_rate": 4.992888225905467e-07,
689
+ "loss": 0.0893,
690
+ "step": 4300
691
+ },
692
+ {
693
+ "epoch": 9.19661733615222,
694
+ "grad_norm": 0.7495366875123604,
695
+ "learning_rate": 3.906443908397017e-07,
696
+ "loss": 0.0873,
697
+ "step": 4350
698
+ },
699
+ {
700
+ "epoch": 9.30232558139535,
701
+ "grad_norm": 0.7691079659468273,
702
+ "learning_rate": 2.9508205842594727e-07,
703
+ "loss": 0.0871,
704
+ "step": 4400
705
+ },
706
+ {
707
+ "epoch": 9.408033826638478,
708
+ "grad_norm": 0.6348077284451538,
709
+ "learning_rate": 2.127319231390168e-07,
710
+ "loss": 0.0877,
711
+ "step": 4450
712
+ },
713
+ {
714
+ "epoch": 9.513742071881607,
715
+ "grad_norm": 0.6760109276931042,
716
+ "learning_rate": 1.4370609578987948e-07,
717
+ "loss": 0.0875,
718
+ "step": 4500
719
+ },
720
+ {
721
+ "epoch": 9.619450317124736,
722
+ "grad_norm": 0.6725906544777629,
723
+ "learning_rate": 8.809854758399017e-08,
724
+ "loss": 0.087,
725
+ "step": 4550
726
+ },
727
+ {
728
+ "epoch": 9.725158562367865,
729
+ "grad_norm": 0.6525914332864488,
730
+ "learning_rate": 4.5984982189636316e-08,
731
+ "loss": 0.0865,
732
+ "step": 4600
733
+ },
734
+ {
735
+ "epoch": 9.830866807610994,
736
+ "grad_norm": 0.6640786754191882,
737
+ "learning_rate": 1.7422732675583808e-08,
738
+ "loss": 0.0888,
739
+ "step": 4650
740
+ },
741
+ {
742
+ "epoch": 9.936575052854122,
743
+ "grad_norm": 0.6669826671643079,
744
+ "learning_rate": 2.4506834582960213e-09,
745
+ "loss": 0.0879,
746
+ "step": 4700
747
+ },
748
+ {
749
+ "epoch": 10.0,
750
+ "eval_loss": 2.3936548233032227,
751
+ "eval_runtime": 18.4489,
752
+ "eval_samples_per_second": 183.751,
753
+ "eval_steps_per_second": 1.464,
754
+ "step": 4730
755
+ },
756
+ {
757
+ "epoch": 10.0,
758
+ "step": 4730,
759
+ "total_flos": 651355949629440.0,
760
+ "train_loss": 0.5446855222952038,
761
+ "train_runtime": 13348.3684,
762
+ "train_samples_per_second": 45.308,
763
+ "train_steps_per_second": 0.354
764
+ }
765
+ ],
766
+ "logging_steps": 50,
767
+ "max_steps": 4730,
768
+ "num_input_tokens_seen": 0,
769
+ "num_train_epochs": 10,
770
+ "save_steps": 500,
771
+ "stateful_callbacks": {
772
+ "TrainerControl": {
773
+ "args": {
774
+ "should_epoch_stop": false,
775
+ "should_evaluate": false,
776
+ "should_log": false,
777
+ "should_save": true,
778
+ "should_training_stop": true
779
+ },
780
+ "attributes": {}
781
+ }
782
+ },
783
+ "total_flos": 651355949629440.0,
784
+ "train_batch_size": 16,
785
+ "trial_name": null,
786
+ "trial_params": null
787
+ }