Hugging Face's logo

MinhPhuoc
/
Qwen2.5-1.5B-Open-R1-Code-GRPO

Text Generation
Transformers
Safetensors
qwen2
Generated from Trainer
open-r1
trl
grpo
conversational
text-generation-inference
Model card Files
xet
Community
MinhPhuoc commited on
Commit
1ad2387
·
verified ·
1 Parent(s): 2d5da2b

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +8 -0
  3. generation_config.json +11 -0
  4. train_results.json +8 -0
  5. trainer_state.json +323 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen2.5-1.5B-Instruct
3
+ library_name: transformers
4
+ model_name: Qwen2.5-1.5B-Open-R1-Code-GRPO
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - grpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Qwen2.5-1.5B-Open-R1-Code-GRPO
13
+
14
+ This model is a fine-tuned version of [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="MinhPhuoc/Qwen2.5-1.5B-Open-R1-Code-GRPO", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/suzhentxt1/huggingface/runs/sxnjvykm)
31
+
32
+
33
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.18.0
38
+ - Transformers: 4.52.3
39
+ - Pytorch: 2.7.0
40
+ - Datasets: 4.4.1
41
+ - Tokenizers: 0.21.4
42
+
43
+ ## Citations
44
+
45
+ Cite GRPO as:
46
+
47
+ ```bibtex
48
+ @article{zhihong2024deepseekmath,
49
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
50
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
51
+ year = 2024,
52
+ eprint = {arXiv:2402.03300},
53
+ }
54
+
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.00023389634034174379,
4
+ "train_runtime": 1466.2797,
5
+ "train_samples": 35735,
6
+ "train_samples_per_second": 0.082,
7
+ "train_steps_per_second": 0.007
8
+ }
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": 151645,
5
+ "pad_token_id": 151643,
6
+ "repetition_penalty": 1.1,
7
+ "temperature": 0.7,
8
+ "top_k": 20,
9
+ "top_p": 0.8,
10
+ "transformers_version": "4.52.3"
11
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": 0.00023389634034174379,
4
+ "train_runtime": 1466.2797,
5
+ "train_samples": 35735,
6
+ "train_samples_per_second": 0.082,
7
+ "train_steps_per_second": 0.007
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,323 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 0.0008395072092681596,
6
+ "eval_steps": 500,
7
+ "global_step": 10,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "clip_ratio/high_max": 0.0,
14
+ "clip_ratio/high_mean": 0.0,
15
+ "clip_ratio/low_mean": 0.0,
16
+ "clip_ratio/low_min": 0.0,
17
+ "clip_ratio/region_mean": 0.0,
18
+ "completions/clipped_ratio": -2.0,
19
+ "completions/max_length": 928.0,
20
+ "completions/max_terminated_length": 928.0,
21
+ "completions/mean_length": 519.3333740234375,
22
+ "completions/mean_terminated_length": 519.3333740234375,
23
+ "completions/min_length": 306.0,
24
+ "completions/min_terminated_length": 306.0,
25
+ "epoch": 8.395072092681596e-05,
26
+ "frac_reward_zero_std": 0.6666666865348816,
27
+ "grad_norm": 1.2832554578781128,
28
+ "kl": 0.0,
29
+ "learning_rate": 0.0,
30
+ "loss": 0.0,
31
+ "num_tokens": 14500.0,
32
+ "reward": 0.008333333767950535,
33
+ "reward_std": 0.01666666753590107,
34
+ "rewards/code_reward/mean": 0.0,
35
+ "rewards/code_reward/std": 0.0,
36
+ "rewards/format_reward/mean": 0.0833333358168602,
37
+ "rewards/format_reward/std": 0.28867512941360474,
38
+ "step": 1
39
+ },
40
+ {
41
+ "clip_ratio/high_max": 0.0,
42
+ "clip_ratio/high_mean": 0.0,
43
+ "clip_ratio/low_mean": 0.0,
44
+ "clip_ratio/low_min": 0.0,
45
+ "clip_ratio/region_mean": 0.0,
46
+ "completions/clipped_ratio": -2.0,
47
+ "completions/max_length": 925.0,
48
+ "completions/max_terminated_length": 925.0,
49
+ "completions/mean_length": 334.75,
50
+ "completions/mean_terminated_length": 334.75,
51
+ "completions/min_length": 109.0,
52
+ "completions/min_terminated_length": 109.0,
53
+ "epoch": 0.00016790144185363192,
54
+ "frac_reward_zero_std": 1.0,
55
+ "grad_norm": 0.0,
56
+ "kl": 0.0,
57
+ "learning_rate": 5e-06,
58
+ "loss": 0.0,
59
+ "num_tokens": 24765.0,
60
+ "reward": 0.0,
61
+ "reward_std": 0.0,
62
+ "rewards/code_reward/mean": 0.0,
63
+ "rewards/code_reward/std": 0.0,
64
+ "rewards/format_reward/mean": 0.0,
65
+ "rewards/format_reward/std": 0.0,
66
+ "step": 2
67
+ },
68
+ {
69
+ "clip_ratio/high_max": 0.0,
70
+ "clip_ratio/high_mean": 0.0,
71
+ "clip_ratio/low_mean": 0.0,
72
+ "clip_ratio/low_min": 0.0,
73
+ "clip_ratio/region_mean": 0.0,
74
+ "completions/clipped_ratio": -2.0,
75
+ "completions/max_length": 760.0,
76
+ "completions/max_terminated_length": 760.0,
77
+ "completions/mean_length": 443.25,
78
+ "completions/mean_terminated_length": 443.25,
79
+ "completions/min_length": 251.0,
80
+ "completions/min_terminated_length": 251.0,
81
+ "epoch": 0.00025185216278044787,
82
+ "frac_reward_zero_std": 0.6666666865348816,
83
+ "grad_norm": 1.4658262729644775,
84
+ "kl": 0.0008602142333984375,
85
+ "learning_rate": 4.864308396768294e-06,
86
+ "loss": 0.0,
87
+ "num_tokens": 35224.0,
88
+ "reward": 0.008333333767950535,
89
+ "reward_std": 0.01666666753590107,
90
+ "rewards/code_reward/mean": 0.0,
91
+ "rewards/code_reward/std": 0.0,
92
+ "rewards/format_reward/mean": 0.0833333358168602,
93
+ "rewards/format_reward/std": 0.28867512941360474,
94
+ "step": 3
95
+ },
96
+ {
97
+ "clip_ratio/high_max": 0.0,
98
+ "clip_ratio/high_mean": 0.0,
99
+ "clip_ratio/low_mean": 0.0,
100
+ "clip_ratio/low_min": 0.0,
101
+ "clip_ratio/region_mean": 0.0,
102
+ "completions/clipped_ratio": -1.75,
103
+ "completions/max_length": 2048.0,
104
+ "completions/max_terminated_length": 709.0,
105
+ "completions/mean_length": 528.1666870117188,
106
+ "completions/mean_terminated_length": 390.0,
107
+ "completions/min_length": 150.0,
108
+ "completions/min_terminated_length": 150.0,
109
+ "epoch": 0.00033580288370726384,
110
+ "frac_reward_zero_std": 0.6666666865348816,
111
+ "grad_norm": 1.4903650283813477,
112
+ "kl": 0.0033283233642578125,
113
+ "learning_rate": 4.473599997017701e-06,
114
+ "loss": 0.0,
115
+ "num_tokens": 49222.0,
116
+ "reward": 0.008333333767950535,
117
+ "reward_std": 0.01666666753590107,
118
+ "rewards/code_reward/mean": 0.0,
119
+ "rewards/code_reward/std": 0.0,
120
+ "rewards/format_reward/mean": 0.0833333358168602,
121
+ "rewards/format_reward/std": 0.28867512941360474,
122
+ "step": 4
123
+ },
124
+ {
125
+ "clip_ratio/high_max": 0.0,
126
+ "clip_ratio/high_mean": 0.0,
127
+ "clip_ratio/low_mean": 0.0,
128
+ "clip_ratio/low_min": 0.0,
129
+ "clip_ratio/region_mean": 0.0,
130
+ "completions/clipped_ratio": -2.0,
131
+ "completions/max_length": 431.0,
132
+ "completions/max_terminated_length": 431.0,
133
+ "completions/mean_length": 299.3333435058594,
134
+ "completions/mean_terminated_length": 299.3333435058594,
135
+ "completions/min_length": 198.0,
136
+ "completions/min_terminated_length": 198.0,
137
+ "epoch": 0.0004197536046340798,
138
+ "frac_reward_zero_std": 0.0,
139
+ "grad_norm": 3.9364893436431885,
140
+ "kl": 0.005321502685546875,
141
+ "learning_rate": 3.875e-06,
142
+ "loss": 0.0001,
143
+ "num_tokens": 59278.0,
144
+ "reward": 0.05000000447034836,
145
+ "reward_std": 0.05257834494113922,
146
+ "rewards/code_reward/mean": 0.0,
147
+ "rewards/code_reward/std": 0.0,
148
+ "rewards/format_reward/mean": 0.5,
149
+ "rewards/format_reward/std": 0.5222329497337341,
150
+ "step": 5
151
+ },
152
+ {
153
+ "clip_ratio/high_max": 0.0,
154
+ "clip_ratio/high_mean": 0.0,
155
+ "clip_ratio/low_mean": 0.0,
156
+ "clip_ratio/low_min": 0.0,
157
+ "clip_ratio/region_mean": 0.0,
158
+ "completions/clipped_ratio": -2.0,
159
+ "completions/max_length": 700.0,
160
+ "completions/max_terminated_length": 700.0,
161
+ "completions/mean_length": 377.41668701171875,
162
+ "completions/mean_terminated_length": 377.41668701171875,
163
+ "completions/min_length": 125.0,
164
+ "completions/min_terminated_length": 125.0,
165
+ "epoch": 0.0005037043255608957,
166
+ "frac_reward_zero_std": 0.6666666865348816,
167
+ "grad_norm": 2.257307291030884,
168
+ "kl": 0.005146026611328125,
169
+ "learning_rate": 3.140708399750594e-06,
170
+ "loss": 0.0001,
171
+ "num_tokens": 72151.0,
172
+ "reward": 0.05000000447034836,
173
+ "reward_std": 0.019245009869337082,
174
+ "rewards/code_reward/mean": 0.0,
175
+ "rewards/code_reward/std": 0.0,
176
+ "rewards/format_reward/mean": 0.5,
177
+ "rewards/format_reward/std": 0.5222329497337341,
178
+ "step": 6
179
+ },
180
+ {
181
+ "clip_ratio/high_max": 0.0,
182
+ "clip_ratio/high_mean": 0.0,
183
+ "clip_ratio/low_mean": 0.0,
184
+ "clip_ratio/low_min": 0.0,
185
+ "clip_ratio/region_mean": 0.0,
186
+ "completions/clipped_ratio": -2.0,
187
+ "completions/max_length": 755.0,
188
+ "completions/max_terminated_length": 755.0,
189
+ "completions/mean_length": 463.41668701171875,
190
+ "completions/mean_terminated_length": 463.41668701171875,
191
+ "completions/min_length": 235.0,
192
+ "completions/min_terminated_length": 235.0,
193
+ "epoch": 0.0005876550464877118,
194
+ "frac_reward_zero_std": 0.0,
195
+ "grad_norm": 2.7628111839294434,
196
+ "kl": 0.00887298583984375,
197
+ "learning_rate": 2.359291600249407e-06,
198
+ "loss": 0.0001,
199
+ "num_tokens": 84448.0,
200
+ "reward": 0.06666667014360428,
201
+ "reward_std": 0.05257834494113922,
202
+ "rewards/code_reward/mean": 0.0,
203
+ "rewards/code_reward/std": 0.0,
204
+ "rewards/format_reward/mean": 0.6666666865348816,
205
+ "rewards/format_reward/std": 0.4923659861087799,
206
+ "step": 7
207
+ },
208
+ {
209
+ "clip_ratio/high_max": 0.0,
210
+ "clip_ratio/high_mean": 0.0,
211
+ "clip_ratio/low_mean": 0.0,
212
+ "clip_ratio/low_min": 0.0,
213
+ "clip_ratio/region_mean": 0.0,
214
+ "completions/clipped_ratio": -2.0,
215
+ "completions/max_length": 956.0,
216
+ "completions/max_terminated_length": 956.0,
217
+ "completions/mean_length": 340.5,
218
+ "completions/mean_terminated_length": 340.5,
219
+ "completions/min_length": 17.0,
220
+ "completions/min_terminated_length": 17.0,
221
+ "epoch": 0.0006716057674145277,
222
+ "frac_reward_zero_std": 0.0,
223
+ "grad_norm": 12.499695777893066,
224
+ "kl": 0.0798187255859375,
225
+ "learning_rate": 1.6250000000000007e-06,
226
+ "loss": 0.0008,
227
+ "num_tokens": 96646.0,
228
+ "reward": 0.07500000298023224,
229
+ "reward_std": 0.05000000447034836,
230
+ "rewards/code_reward/mean": 0.0,
231
+ "rewards/code_reward/std": 0.0,
232
+ "rewards/format_reward/mean": 0.75,
233
+ "rewards/format_reward/std": 0.45226702094078064,
234
+ "step": 8
235
+ },
236
+ {
237
+ "clip_ratio/high_max": 0.0,
238
+ "clip_ratio/high_mean": 0.0,
239
+ "clip_ratio/low_mean": 0.0,
240
+ "clip_ratio/low_min": 0.0,
241
+ "clip_ratio/region_mean": 0.0,
242
+ "completions/clipped_ratio": -1.75,
243
+ "completions/max_length": 2048.0,
244
+ "completions/max_terminated_length": 1160.0,
245
+ "completions/mean_length": 593.4166870117188,
246
+ "completions/mean_terminated_length": 461.18182373046875,
247
+ "completions/min_length": 208.0,
248
+ "completions/min_terminated_length": 208.0,
249
+ "epoch": 0.0007555564883413436,
250
+ "frac_reward_zero_std": 0.3333333432674408,
251
+ "grad_norm": 2.6419012546539307,
252
+ "kl": 0.03733062744140625,
253
+ "learning_rate": 1.0264000029823e-06,
254
+ "loss": 0.0004,
255
+ "num_tokens": 111479.0,
256
+ "reward": 0.05000000447034836,
257
+ "reward_std": 0.03333333507180214,
258
+ "rewards/code_reward/mean": 0.0,
259
+ "rewards/code_reward/std": 0.0,
260
+ "rewards/format_reward/mean": 0.5,
261
+ "rewards/format_reward/std": 0.5222329497337341,
262
+ "step": 9
263
+ },
264
+ {
265
+ "clip_ratio/high_max": 0.0,
266
+ "clip_ratio/high_mean": 0.0,
267
+ "clip_ratio/low_mean": 0.0,
268
+ "clip_ratio/low_min": 0.0,
269
+ "clip_ratio/region_mean": 0.0,
270
+ "completions/clipped_ratio": -2.0,
271
+ "completions/max_length": 677.0,
272
+ "completions/max_terminated_length": 677.0,
273
+ "completions/mean_length": 376.41668701171875,
274
+ "completions/mean_terminated_length": 376.41668701171875,
275
+ "completions/min_length": 160.0,
276
+ "completions/min_terminated_length": 160.0,
277
+ "epoch": 0.0008395072092681596,
278
+ "frac_reward_zero_std": 0.3333333432674408,
279
+ "grad_norm": 14.455758094787598,
280
+ "kl": 0.0934295654296875,
281
+ "learning_rate": 6.356916032317063e-07,
282
+ "loss": 0.0009,
283
+ "num_tokens": 122756.0,
284
+ "reward": 0.06666667014360428,
285
+ "reward_std": 0.038490019738674164,
286
+ "rewards/code_reward/mean": 0.0,
287
+ "rewards/code_reward/std": 0.0,
288
+ "rewards/format_reward/mean": 0.6666666865348816,
289
+ "rewards/format_reward/std": 0.4923659861087799,
290
+ "step": 10
291
+ },
292
+ {
293
+ "epoch": 0.0008395072092681596,
294
+ "step": 10,
295
+ "total_flos": 0.0,
296
+ "train_loss": 0.00023389634034174379,
297
+ "train_runtime": 1466.2797,
298
+ "train_samples_per_second": 0.082,
299
+ "train_steps_per_second": 0.007
300
+ }
301
+ ],
302
+ "logging_steps": 1.0,
303
+ "max_steps": 10,
304
+ "num_input_tokens_seen": 122756,
305
+ "num_train_epochs": 1,
306
+ "save_steps": 50,
307
+ "stateful_callbacks": {
308
+ "TrainerControl": {
309
+ "args": {
310
+ "should_epoch_stop": false,
311
+ "should_evaluate": false,
312
+ "should_log": false,
313
+ "should_save": true,
314
+ "should_training_stop": true
315
+ },
316
+ "attributes": {}
317
+ }
318
+ },
319
+ "total_flos": 0.0,
320
+ "train_batch_size": 1,
321
+ "trial_name": null,
322
+ "trial_params": null
323
+ }