Hugging Face's logo

Blancy
/
Qwen3-1.7B-Open-R1-Code-GRPO

Text Generation
Transformers
Safetensors
qwen3
Generated from Trainer
open-r1
trl
grpo
conversational
text-generation-inference
Model card Files
xet
Community
Blancy commited on
Commit
88fb2fe
·
verified ·
1 Parent(s): 4706082

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +8 -0
  3. generation_config.json +13 -0
  4. train_results.json +8 -0
  5. trainer_state.json +550 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-1.7B
3
+ library_name: transformers
4
+ model_name: Qwen3-1.7B-Open-R1-Code-GRPO
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - grpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Qwen3-1.7B-Open-R1-Code-GRPO
13
+
14
+ This model is a fine-tuned version of [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="Blancy/Qwen3-1.7B-Open-R1-Code-GRPO", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+
31
+
32
+
33
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.15.2
38
+ - Transformers: 4.52.3
39
+ - Pytorch: 2.5.1
40
+ - Datasets: 3.6.0
41
+ - Tokenizers: 0.21.1
42
+
43
+ ## Citations
44
+
45
+ Cite GRPO as:
46
+
47
+ ```bibtex
48
+ @article{zhihong2024deepseekmath,
49
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
50
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
51
+ year = 2024,
52
+ eprint = {arXiv:2402.03300},
53
+ }
54
+
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.024176058914698324,
4
+ "train_runtime": 19938.5063,
5
+ "train_samples": 1086,
6
+ "train_samples_per_second": 0.054,
7
+ "train_steps_per_second": 0.002
8
+ }
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 151645,
6
+ 151643
7
+ ],
8
+ "pad_token_id": 151643,
9
+ "temperature": 0.6,
10
+ "top_k": 20,
11
+ "top_p": 0.95,
12
+ "transformers_version": "4.52.3"
13
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.024176058914698324,
4
+ "train_runtime": 19938.5063,
5
+ "train_samples": 1086,
6
+ "train_samples_per_second": 0.054,
7
+ "train_steps_per_second": 0.002
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,550 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.0,
6
+ "eval_steps": 500,
7
+ "global_step": 39,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "completion_length": 3541.9607543945312,
14
+ "epoch": 0.02564102564102564,
15
+ "grad_norm": 5.551374912261963,
16
+ "kl": 0.0,
17
+ "learning_rate": 0.0,
18
+ "loss": 0.0,
19
+ "reward": 0.3934539742767811,
20
+ "reward_std": 0.3425147756934166,
21
+ "rewards/code_reward": 0.3934539742767811,
22
+ "rewards/format_reward": 0.0,
23
+ "step": 1
24
+ },
25
+ {
26
+ "completion_length": 3543.0106201171875,
27
+ "epoch": 0.05128205128205128,
28
+ "grad_norm": 1.4287736415863037,
29
+ "kl": 0.0,
30
+ "learning_rate": 5e-06,
31
+ "loss": 0.0,
32
+ "reward": 0.46500000543892384,
33
+ "reward_std": 0.19661623612046242,
34
+ "rewards/code_reward": 0.46499999053776264,
35
+ "rewards/format_reward": 0.0,
36
+ "step": 2
37
+ },
38
+ {
39
+ "completion_length": 3270.2929077148438,
40
+ "epoch": 0.07692307692307693,
41
+ "grad_norm": 2.6348750591278076,
42
+ "kl": 0.00787353515625,
43
+ "learning_rate": 1e-05,
44
+ "loss": 0.0003,
45
+ "reward": 0.5235317498445511,
46
+ "reward_std": 0.30637360364198685,
47
+ "rewards/code_reward": 0.5235317498445511,
48
+ "rewards/format_reward": 0.0,
49
+ "step": 3
50
+ },
51
+ {
52
+ "completion_length": 2943.4500122070312,
53
+ "epoch": 0.10256410256410256,
54
+ "grad_norm": 0.7752403020858765,
55
+ "kl": 0.0511474609375,
56
+ "learning_rate": 9.98378869844137e-06,
57
+ "loss": 0.002,
58
+ "reward": 0.5904354751110077,
59
+ "reward_std": 0.21103981602936983,
60
+ "rewards/code_reward": 0.5904354676604271,
61
+ "rewards/format_reward": 0.0,
62
+ "step": 4
63
+ },
64
+ {
65
+ "completion_length": 3438.1785888671875,
66
+ "epoch": 0.1282051282051282,
67
+ "grad_norm": 3.3206803798675537,
68
+ "kl": 0.120361328125,
69
+ "learning_rate": 9.935271596564688e-06,
70
+ "loss": 0.0048,
71
+ "reward": 0.47174738347530365,
72
+ "reward_std": 0.1500780526548624,
73
+ "rewards/code_reward": 0.47174738347530365,
74
+ "rewards/format_reward": 0.0,
75
+ "step": 5
76
+ },
77
+ {
78
+ "completion_length": 3396.6856689453125,
79
+ "epoch": 0.15384615384615385,
80
+ "grad_norm": 0.2983720600605011,
81
+ "kl": 0.2265625,
82
+ "learning_rate": 9.854798261200746e-06,
83
+ "loss": 0.0091,
84
+ "reward": 0.5083690956234932,
85
+ "reward_std": 0.23467476293444633,
86
+ "rewards/code_reward": 0.5083690956234932,
87
+ "rewards/format_reward": 0.0,
88
+ "step": 6
89
+ },
90
+ {
91
+ "completion_length": 3636.0071411132812,
92
+ "epoch": 0.1794871794871795,
93
+ "grad_norm": 0.29332682490348816,
94
+ "kl": 0.26611328125,
95
+ "learning_rate": 9.74294850457488e-06,
96
+ "loss": 0.0106,
97
+ "reward": 0.387291356921196,
98
+ "reward_std": 0.23526490107178688,
99
+ "rewards/code_reward": 0.387291356921196,
100
+ "rewards/format_reward": 0.0,
101
+ "step": 7
102
+ },
103
+ {
104
+ "completion_length": 3375.800048828125,
105
+ "epoch": 0.20512820512820512,
106
+ "grad_norm": 0.3416775166988373,
107
+ "kl": 0.3359375,
108
+ "learning_rate": 9.600528206746613e-06,
109
+ "loss": 0.0134,
110
+ "reward": 0.48297154158353806,
111
+ "reward_std": 0.25456428155303,
112
+ "rewards/code_reward": 0.48297156393527985,
113
+ "rewards/format_reward": 0.0,
114
+ "step": 8
115
+ },
116
+ {
117
+ "completion_length": 3155.9608154296875,
118
+ "epoch": 0.23076923076923078,
119
+ "grad_norm": 0.240010604262352,
120
+ "kl": 0.3779296875,
121
+ "learning_rate": 9.428563509225348e-06,
122
+ "loss": 0.0151,
123
+ "reward": 0.5900027677416801,
124
+ "reward_std": 0.18181878328323364,
125
+ "rewards/code_reward": 0.5900027677416801,
126
+ "rewards/format_reward": 0.0,
127
+ "step": 9
128
+ },
129
+ {
130
+ "completion_length": 3539.346435546875,
131
+ "epoch": 0.2564102564102564,
132
+ "grad_norm": 0.23563309013843536,
133
+ "kl": 0.3955078125,
134
+ "learning_rate": 9.22829342159729e-06,
135
+ "loss": 0.0158,
136
+ "reward": 0.4020918384194374,
137
+ "reward_std": 0.22648156061768532,
138
+ "rewards/code_reward": 0.4020918384194374,
139
+ "rewards/format_reward": 0.0,
140
+ "step": 10
141
+ },
142
+ {
143
+ "completion_length": 3615.935791015625,
144
+ "epoch": 0.28205128205128205,
145
+ "grad_norm": 0.19946347177028656,
146
+ "kl": 0.47216796875,
147
+ "learning_rate": 9.001160894432979e-06,
148
+ "loss": 0.0189,
149
+ "reward": 0.41117217019200325,
150
+ "reward_std": 0.20069691445678473,
151
+ "rewards/code_reward": 0.41117217019200325,
152
+ "rewards/format_reward": 0.0,
153
+ "step": 11
154
+ },
155
+ {
156
+ "completion_length": 3652.20361328125,
157
+ "epoch": 0.3076923076923077,
158
+ "grad_norm": 0.21409198641777039,
159
+ "kl": 0.556640625,
160
+ "learning_rate": 8.748802422795361e-06,
161
+ "loss": 0.0222,
162
+ "reward": 0.41303257271647453,
163
+ "reward_std": 0.2046195026487112,
164
+ "rewards/code_reward": 0.41303258016705513,
165
+ "rewards/format_reward": 0.0,
166
+ "step": 12
167
+ },
168
+ {
169
+ "completion_length": 3682.6357421875,
170
+ "epoch": 0.3333333333333333,
171
+ "grad_norm": 0.6496288180351257,
172
+ "kl": 0.580078125,
173
+ "learning_rate": 8.473036255255368e-06,
174
+ "loss": 0.0232,
175
+ "reward": 0.4388655610382557,
176
+ "reward_std": 0.22819811291992664,
177
+ "rewards/code_reward": 0.4388655610382557,
178
+ "rewards/format_reward": 0.0,
179
+ "step": 13
180
+ },
181
+ {
182
+ "completion_length": 3709.0142211914062,
183
+ "epoch": 0.358974358974359,
184
+ "grad_norm": 0.23685197532176971,
185
+ "kl": 0.6474609375,
186
+ "learning_rate": 8.175849293369292e-06,
187
+ "loss": 0.0259,
188
+ "reward": 0.37063881754875183,
189
+ "reward_std": 0.23854901269078255,
190
+ "rewards/code_reward": 0.37063881754875183,
191
+ "rewards/format_reward": 0.0,
192
+ "step": 14
193
+ },
194
+ {
195
+ "completion_length": 3758.8857421875,
196
+ "epoch": 0.38461538461538464,
197
+ "grad_norm": 0.22902953624725342,
198
+ "kl": 0.66015625,
199
+ "learning_rate": 7.859382776007544e-06,
200
+ "loss": 0.0264,
201
+ "reward": 0.31185516342520714,
202
+ "reward_std": 0.22547182254493237,
203
+ "rewards/code_reward": 0.31185516342520714,
204
+ "rewards/format_reward": 0.0,
205
+ "step": 15
206
+ },
207
+ {
208
+ "completion_length": 3474.4249877929688,
209
+ "epoch": 0.41025641025641024,
210
+ "grad_norm": 0.2739468514919281,
211
+ "kl": 0.658203125,
212
+ "learning_rate": 7.52591685167953e-06,
213
+ "loss": 0.0263,
214
+ "reward": 0.46756455302238464,
215
+ "reward_std": 0.21291000582277775,
216
+ "rewards/code_reward": 0.46756456792354584,
217
+ "rewards/format_reward": 0.0,
218
+ "step": 16
219
+ },
220
+ {
221
+ "completion_length": 3608.0107421875,
222
+ "epoch": 0.4358974358974359,
223
+ "grad_norm": 0.26821818947792053,
224
+ "kl": 0.66796875,
225
+ "learning_rate": 7.1778541500113895e-06,
226
+ "loss": 0.0267,
227
+ "reward": 0.33532585576176643,
228
+ "reward_std": 0.29637256264686584,
229
+ "rewards/code_reward": 0.33532586693763733,
230
+ "rewards/format_reward": 0.0,
231
+ "step": 17
232
+ },
233
+ {
234
+ "completion_length": 3718.9964599609375,
235
+ "epoch": 0.46153846153846156,
236
+ "grad_norm": 0.24430078268051147,
237
+ "kl": 0.6259765625,
238
+ "learning_rate": 6.817702470744477e-06,
239
+ "loss": 0.025,
240
+ "reward": 0.22364513762295246,
241
+ "reward_std": 0.23529299348592758,
242
+ "rewards/code_reward": 0.22364513762295246,
243
+ "rewards/format_reward": 0.0,
244
+ "step": 18
245
+ },
246
+ {
247
+ "completion_length": 3312.9071044921875,
248
+ "epoch": 0.48717948717948717,
249
+ "grad_norm": 0.2842939794063568,
250
+ "kl": 0.5859375,
251
+ "learning_rate": 6.448056714980768e-06,
252
+ "loss": 0.0234,
253
+ "reward": 0.46438145264983177,
254
+ "reward_std": 0.2700807861983776,
255
+ "rewards/code_reward": 0.46438145637512207,
256
+ "rewards/format_reward": 0.0,
257
+ "step": 19
258
+ },
259
+ {
260
+ "completion_length": 3342.6749877929688,
261
+ "epoch": 0.5128205128205128,
262
+ "grad_norm": 0.2545294761657715,
263
+ "kl": 0.5576171875,
264
+ "learning_rate": 6.071580188860955e-06,
265
+ "loss": 0.0223,
266
+ "reward": 0.5150031447410583,
267
+ "reward_std": 0.2922932505607605,
268
+ "rewards/code_reward": 0.5150031298398972,
269
+ "rewards/format_reward": 0.0,
270
+ "step": 20
271
+ },
272
+ {
273
+ "completion_length": 2976.3214111328125,
274
+ "epoch": 0.5384615384615384,
275
+ "grad_norm": 0.23293468356132507,
276
+ "kl": 0.6181640625,
277
+ "learning_rate": 5.690985414382668e-06,
278
+ "loss": 0.0247,
279
+ "reward": 0.46903225034475327,
280
+ "reward_std": 0.23096787184476852,
281
+ "rewards/code_reward": 0.4690322279930115,
282
+ "rewards/format_reward": 0.0,
283
+ "step": 21
284
+ },
285
+ {
286
+ "completion_length": 3359.9892578125,
287
+ "epoch": 0.5641025641025641,
288
+ "grad_norm": 0.2094847410917282,
289
+ "kl": 0.5703125,
290
+ "learning_rate": 5.309014585617335e-06,
291
+ "loss": 0.0228,
292
+ "reward": 0.40152605809271336,
293
+ "reward_std": 0.2346283309161663,
294
+ "rewards/code_reward": 0.40152604319155216,
295
+ "rewards/format_reward": 0.0,
296
+ "step": 22
297
+ },
298
+ {
299
+ "completion_length": 2678.5321655273438,
300
+ "epoch": 0.5897435897435898,
301
+ "grad_norm": 0.24114222824573517,
302
+ "kl": 0.5654296875,
303
+ "learning_rate": 4.928419811139046e-06,
304
+ "loss": 0.0227,
305
+ "reward": 0.608155146241188,
306
+ "reward_std": 0.14721081405878067,
307
+ "rewards/code_reward": 0.6081551611423492,
308
+ "rewards/format_reward": 0.0,
309
+ "step": 23
310
+ },
311
+ {
312
+ "completion_length": 2912.8821411132812,
313
+ "epoch": 0.6153846153846154,
314
+ "grad_norm": 0.3605176508426666,
315
+ "kl": 0.5908203125,
316
+ "learning_rate": 4.551943285019233e-06,
317
+ "loss": 0.0236,
318
+ "reward": 0.5329480394721031,
319
+ "reward_std": 0.1990287434309721,
320
+ "rewards/code_reward": 0.5329480245709419,
321
+ "rewards/format_reward": 0.0,
322
+ "step": 24
323
+ },
324
+ {
325
+ "completion_length": 3301.3857421875,
326
+ "epoch": 0.6410256410256411,
327
+ "grad_norm": 6.35435152053833,
328
+ "kl": 0.716796875,
329
+ "learning_rate": 4.182297529255525e-06,
330
+ "loss": 0.0287,
331
+ "reward": 0.36159154400229454,
332
+ "reward_std": 0.2206190638244152,
333
+ "rewards/code_reward": 0.36159154027700424,
334
+ "rewards/format_reward": 0.0,
335
+ "step": 25
336
+ },
337
+ {
338
+ "completion_length": 3339.2214965820312,
339
+ "epoch": 0.6666666666666666,
340
+ "grad_norm": 0.4265838861465454,
341
+ "kl": 0.7275390625,
342
+ "learning_rate": 3.822145849988612e-06,
343
+ "loss": 0.0291,
344
+ "reward": 0.3386395741254091,
345
+ "reward_std": 0.18014118261635303,
346
+ "rewards/code_reward": 0.3386395741254091,
347
+ "rewards/format_reward": 0.0,
348
+ "step": 26
349
+ },
350
+ {
351
+ "completion_length": 3056.1214599609375,
352
+ "epoch": 0.6923076923076923,
353
+ "grad_norm": 0.4635300636291504,
354
+ "kl": 0.7158203125,
355
+ "learning_rate": 3.4740831483204696e-06,
356
+ "loss": 0.0286,
357
+ "reward": 0.4216299280524254,
358
+ "reward_std": 0.24011335149407387,
359
+ "rewards/code_reward": 0.4216299429535866,
360
+ "rewards/format_reward": 0.0,
361
+ "step": 27
362
+ },
363
+ {
364
+ "completion_length": 2741.1821899414062,
365
+ "epoch": 0.717948717948718,
366
+ "grad_norm": 0.26547616720199585,
367
+ "kl": 0.6884765625,
368
+ "learning_rate": 3.1406172239924583e-06,
369
+ "loss": 0.0275,
370
+ "reward": 0.5913942456245422,
371
+ "reward_std": 0.19475560076534748,
372
+ "rewards/code_reward": 0.5913942456245422,
373
+ "rewards/format_reward": 0.0,
374
+ "step": 28
375
+ },
376
+ {
377
+ "completion_length": 3502.3857421875,
378
+ "epoch": 0.7435897435897436,
379
+ "grad_norm": 0.3135121464729309,
380
+ "kl": 0.8955078125,
381
+ "learning_rate": 2.8241507066307106e-06,
382
+ "loss": 0.0358,
383
+ "reward": 0.33309811167418957,
384
+ "reward_std": 0.2021036557853222,
385
+ "rewards/code_reward": 0.3330980967730284,
386
+ "rewards/format_reward": 0.0,
387
+ "step": 29
388
+ },
389
+ {
390
+ "completion_length": 3053.2321166992188,
391
+ "epoch": 0.7692307692307693,
392
+ "grad_norm": 0.25711485743522644,
393
+ "kl": 0.880859375,
394
+ "learning_rate": 2.526963744744635e-06,
395
+ "loss": 0.0352,
396
+ "reward": 0.44350775331258774,
397
+ "reward_std": 0.19116137735545635,
398
+ "rewards/code_reward": 0.44350775331258774,
399
+ "rewards/format_reward": 0.0,
400
+ "step": 30
401
+ },
402
+ {
403
+ "completion_length": 3173.0178833007812,
404
+ "epoch": 0.7948717948717948,
405
+ "grad_norm": 0.3827289044857025,
406
+ "kl": 0.8798828125,
407
+ "learning_rate": 2.2511975772046403e-06,
408
+ "loss": 0.0352,
409
+ "reward": 0.42228348553180695,
410
+ "reward_std": 0.205778568983078,
411
+ "rewards/code_reward": 0.42228347808122635,
412
+ "rewards/format_reward": 0.0,
413
+ "step": 31
414
+ },
415
+ {
416
+ "completion_length": 3241.3535766601562,
417
+ "epoch": 0.8205128205128205,
418
+ "grad_norm": 0.4431661367416382,
419
+ "kl": 0.9033203125,
420
+ "learning_rate": 1.9988391055670234e-06,
421
+ "loss": 0.0362,
422
+ "reward": 0.3359471336007118,
423
+ "reward_std": 0.20435638166964054,
424
+ "rewards/code_reward": 0.3359471336007118,
425
+ "rewards/format_reward": 0.0,
426
+ "step": 32
427
+ },
428
+ {
429
+ "completion_length": 2643.057159423828,
430
+ "epoch": 0.8461538461538461,
431
+ "grad_norm": 0.5116384029388428,
432
+ "kl": 0.8017578125,
433
+ "learning_rate": 1.771706578402711e-06,
434
+ "loss": 0.0321,
435
+ "reward": 0.5609605759382248,
436
+ "reward_std": 0.18299106322228909,
437
+ "rewards/code_reward": 0.5609605610370636,
438
+ "rewards/format_reward": 0.0,
439
+ "step": 33
440
+ },
441
+ {
442
+ "completion_length": 2872.3214721679688,
443
+ "epoch": 0.8717948717948718,
444
+ "grad_norm": 0.44135168194770813,
445
+ "kl": 0.962890625,
446
+ "learning_rate": 1.5714364907746535e-06,
447
+ "loss": 0.0385,
448
+ "reward": 0.41167889907956123,
449
+ "reward_std": 0.1887526996433735,
450
+ "rewards/code_reward": 0.41167885810136795,
451
+ "rewards/format_reward": 0.0,
452
+ "step": 34
453
+ },
454
+ {
455
+ "completion_length": 2978.9928588867188,
456
+ "epoch": 0.8974358974358975,
457
+ "grad_norm": 0.6080113053321838,
458
+ "kl": 1.0380859375,
459
+ "learning_rate": 1.399471793253389e-06,
460
+ "loss": 0.0416,
461
+ "reward": 0.3960940055549145,
462
+ "reward_std": 0.20829082280397415,
463
+ "rewards/code_reward": 0.3960940055549145,
464
+ "rewards/format_reward": 0.0,
465
+ "step": 35
466
+ },
467
+ {
468
+ "completion_length": 3258.403564453125,
469
+ "epoch": 0.9230769230769231,
470
+ "grad_norm": 0.41639065742492676,
471
+ "kl": 1.072265625,
472
+ "learning_rate": 1.257051495425121e-06,
473
+ "loss": 0.0429,
474
+ "reward": 0.1503874734044075,
475
+ "reward_std": 0.12068512104451656,
476
+ "rewards/code_reward": 0.15038747526705265,
477
+ "rewards/format_reward": 0.0,
478
+ "step": 36
479
+ },
480
+ {
481
+ "completion_length": 2731.4713745117188,
482
+ "epoch": 0.9487179487179487,
483
+ "grad_norm": 0.33002978563308716,
484
+ "kl": 1.0263671875,
485
+ "learning_rate": 1.1452017387992552e-06,
486
+ "loss": 0.041,
487
+ "reward": 0.234993327409029,
488
+ "reward_std": 0.1311760265380144,
489
+ "rewards/code_reward": 0.234993327409029,
490
+ "rewards/format_reward": 0.0,
491
+ "step": 37
492
+ },
493
+ {
494
+ "completion_length": 2790.7571411132812,
495
+ "epoch": 0.9743589743589743,
496
+ "grad_norm": 0.3518536388874054,
497
+ "kl": 1.005859375,
498
+ "learning_rate": 1.0647284034353122e-06,
499
+ "loss": 0.0402,
500
+ "reward": 0.28501298278570175,
501
+ "reward_std": 0.07613634853623807,
502
+ "rewards/code_reward": 0.28501297906041145,
503
+ "rewards/format_reward": 0.0,
504
+ "step": 38
505
+ },
506
+ {
507
+ "completion_length": 2785.1597290039062,
508
+ "epoch": 1.0,
509
+ "grad_norm": 0.6018698215484619,
510
+ "kl": 1.119140625,
511
+ "learning_rate": 1.0162113015586309e-06,
512
+ "loss": 0.0448,
513
+ "reward": 0.26603568717837334,
514
+ "reward_std": 0.09410964651033282,
515
+ "rewards/code_reward": 0.26603569462895393,
516
+ "rewards/format_reward": 0.0,
517
+ "step": 39
518
+ },
519
+ {
520
+ "epoch": 1.0,
521
+ "step": 39,
522
+ "total_flos": 0.0,
523
+ "train_loss": 0.024176058914698324,
524
+ "train_runtime": 19938.5063,
525
+ "train_samples_per_second": 0.054,
526
+ "train_steps_per_second": 0.002
527
+ }
528
+ ],
529
+ "logging_steps": 1,
530
+ "max_steps": 39,
531
+ "num_input_tokens_seen": 0,
532
+ "num_train_epochs": 1,
533
+ "save_steps": 500,
534
+ "stateful_callbacks": {
535
+ "TrainerControl": {
536
+ "args": {
537
+ "should_epoch_stop": false,
538
+ "should_evaluate": false,
539
+ "should_log": false,
540
+ "should_save": true,
541
+ "should_training_stop": true
542
+ },
543
+ "attributes": {}
544
+ }
545
+ },
546
+ "total_flos": 0.0,
547
+ "train_batch_size": 10,
548
+ "trial_name": null,
549
+ "trial_params": null
550
+ }