Hugging Face's logo

yarin-shaked
/
Qwen3-Codeforces-GRPO

Text Generation
Transformers
Safetensors
qwen3
Generated from Trainer
open-r1
trl
grpo
conversational
text-generation-inference
Model card Files
xet
 Community
yarin-shaked commited on
Commit
03cf9e7
·
verified ·
1 Parent(s): 8f5b74c

Model save

Browse files
Files changed (5) hide show
  1. README.md +68 -0
  2. all_results.json +8 -0
  3. generation_config.json +10 -0
  4. train_results.json +8 -0
  5. trainer_state.json +259 -0
README.md ADDED
@@ -0,0 +1,68 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: Qwen/Qwen3-0.6B
3
+ library_name: transformers
4
+ model_name: Qwen3-Codeforces-GRPO
5
+ tags:
6
+ - generated_from_trainer
7
+ - trl
8
+ - grpo
9
+ licence: license
10
+ ---
11
+
12
+ # Model Card for Qwen3-Codeforces-GRPO
13
+
14
+ This model is a fine-tuned version of [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B).
15
+ It has been trained using [TRL](https://github.com/huggingface/trl).
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ from transformers import pipeline
21
+
22
+ question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
23
+ generator = pipeline("text-generation", model="yarin-shaked/Qwen3-Codeforces-GRPO", device="cuda")
24
+ output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
25
+ print(output["generated_text"])
26
+ ```
27
+
28
+ ## Training procedure
29
+
30
+ [<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/yarinshaked/open-r1/runs/jam073ra)
31
+
32
+
33
+ This model was trained with GRPO, a method introduced in [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://huggingface.co/papers/2402.03300).
34
+
35
+ ### Framework versions
36
+
37
+ - TRL: 0.18.0
38
+ - Transformers: 4.52.3
39
+ - Pytorch: 2.6.0
40
+ - Datasets: 4.2.0
41
+ - Tokenizers: 0.21.4
42
+
43
+ ## Citations
44
+
45
+ Cite GRPO as:
46
+
47
+ ```bibtex
48
+ @article{zhihong2024deepseekmath,
49
+ title = {{DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models}},
50
+ author = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
51
+ year = 2024,
52
+ eprint = {arXiv:2402.03300},
53
+ }
54
+
55
+ ```
56
+
57
+ Cite TRL as:
58
+
59
+ ```bibtex
60
+ @misc{vonwerra2022trl,
61
+ title = {{TRL: Transformer Reinforcement Learning}},
62
+ author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
63
+ year = 2020,
64
+ journal = {GitHub repository},
65
+ publisher = {GitHub},
66
+ howpublished = {\url{https://github.com/huggingface/trl}}
67
+ }
68
+ ```
all_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": -0.039116610772907734,
4
+ "train_runtime": 615.0598,
5
+ "train_samples": 16676,
6
+ "train_samples_per_second": 0.052,
7
+ "train_steps_per_second": 0.013
8
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": 151645,
5
+ "pad_token_id": 151643,
6
+ "temperature": 0.6,
7
+ "top_k": 20,
8
+ "top_p": 0.95,
9
+ "transformers_version": "4.52.3"
10
+ }
train_results.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "total_flos": 0.0,
3
+ "train_loss": -0.039116610772907734,
4
+ "train_runtime": 615.0598,
5
+ "train_samples": 16676,
6
+ "train_samples_per_second": 0.052,
7
+ "train_steps_per_second": 0.013
8
+ }
trainer_state.json ADDED
@@ -0,0 +1,259 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_global_step": null,
3
+ "best_metric": null,
4
+ "best_model_checkpoint": null,
5
+ "epoch": 1.0,
6
+ "eval_steps": 500,
7
+ "global_step": 8,
8
+ "is_hyper_param_search": false,
9
+ "is_local_process_zero": true,
10
+ "is_world_process_zero": true,
11
+ "log_history": [
12
+ {
13
+ "clip_ratio/high_max": 0.0,
14
+ "clip_ratio/high_mean": 0.0,
15
+ "clip_ratio/low_mean": 0.0,
16
+ "clip_ratio/low_min": 0.0,
17
+ "clip_ratio/region_mean": 0.0,
18
+ "completions/clipped_ratio": -0.25,
19
+ "completions/max_length": 8192.0,
20
+ "completions/max_terminated_length": 8050.0,
21
+ "completions/mean_length": 6862.1875,
22
+ "completions/mean_terminated_length": 3936.60009765625,
23
+ "completions/min_length": 1679.0,
24
+ "completions/min_terminated_length": 1679.0,
25
+ "epoch": 0.125,
26
+ "frac_reward_zero_std": 0.5,
27
+ "grad_norm": 0.1540709645657331,
28
+ "learning_rate": 0.0,
29
+ "loss": -0.0397,
30
+ "num_tokens": 31407.0,
31
+ "reward": -0.699999988079071,
32
+ "reward_std": 0.125,
33
+ "rewards/cf_code_reward/mean": -0.699999988079071,
34
+ "rewards/cf_code_reward/std": 0.46043458580970764,
35
+ "rewards/code_format_reward/mean": 0.0,
36
+ "rewards/code_format_reward/std": 0.0,
37
+ "step": 1
38
+ },
39
+ {
40
+ "clip_ratio/high_max": NaN,
41
+ "clip_ratio/high_mean": NaN,
42
+ "clip_ratio/low_mean": NaN,
43
+ "clip_ratio/low_min": NaN,
44
+ "clip_ratio/region_mean": NaN,
45
+ "completions/clipped_ratio": 1.0,
46
+ "completions/max_length": 8192.0,
47
+ "completions/max_terminated_length": 0.0,
48
+ "completions/mean_length": 8192.0,
49
+ "completions/mean_terminated_length": 0.0,
50
+ "completions/min_length": 8192.0,
51
+ "completions/min_terminated_length": 0.0,
52
+ "epoch": 0.25,
53
+ "frac_reward_zero_std": 1.0,
54
+ "grad_norm": 0.0,
55
+ "learning_rate": 1e-06,
56
+ "loss": 0.0,
57
+ "num_tokens": 42723.0,
58
+ "reward": -1.0,
59
+ "reward_std": 0.0,
60
+ "rewards/cf_code_reward/mean": -1.0,
61
+ "rewards/cf_code_reward/std": 0.0,
62
+ "rewards/code_format_reward/mean": 0.0,
63
+ "rewards/code_format_reward/std": 0.0,
64
+ "step": 2
65
+ },
66
+ {
67
+ "clip_ratio/high_max": 0.0,
68
+ "clip_ratio/high_mean": 0.0,
69
+ "clip_ratio/low_mean": 0.0,
70
+ "clip_ratio/low_min": 0.0,
71
+ "clip_ratio/region_mean": 0.0,
72
+ "completions/clipped_ratio": 0.75,
73
+ "completions/max_length": 8192.0,
74
+ "completions/max_terminated_length": 7057.0,
75
+ "completions/mean_length": 8121.0625,
76
+ "completions/mean_terminated_length": 7057.0,
77
+ "completions/min_length": 7057.0,
78
+ "completions/min_terminated_length": 7057.0,
79
+ "epoch": 0.375,
80
+ "frac_reward_zero_std": 0.75,
81
+ "grad_norm": 0.16390367065221123,
82
+ "learning_rate": 1e-06,
83
+ "loss": -0.0404,
84
+ "num_tokens": 62168.0,
85
+ "reward": -0.9375,
86
+ "reward_std": 0.125,
87
+ "rewards/cf_code_reward/mean": -0.9375,
88
+ "rewards/cf_code_reward/std": 0.25,
89
+ "rewards/code_format_reward/mean": 0.0,
90
+ "rewards/code_format_reward/std": 0.0,
91
+ "step": 3
92
+ },
93
+ {
94
+ "clip_ratio/high_max": 0.0,
95
+ "clip_ratio/high_mean": 0.0,
96
+ "clip_ratio/low_mean": 0.0,
97
+ "clip_ratio/low_min": 0.0,
98
+ "clip_ratio/region_mean": 0.0,
99
+ "completions/clipped_ratio": 0.25,
100
+ "completions/max_length": 8192.0,
101
+ "completions/max_terminated_length": 7780.0,
102
+ "completions/mean_length": 7816.625,
103
+ "completions/mean_terminated_length": 6190.0,
104
+ "completions/min_length": 4315.0,
105
+ "completions/min_terminated_length": 4315.0,
106
+ "epoch": 0.5,
107
+ "frac_reward_zero_std": 0.75,
108
+ "grad_norm": 0.2878601922045561,
109
+ "learning_rate": 1e-06,
110
+ "loss": -0.0875,
111
+ "num_tokens": 91526.0,
112
+ "reward": -0.8125,
113
+ "reward_std": 0.248746857047081,
114
+ "rewards/cf_code_reward/mean": -0.8125,
115
+ "rewards/cf_code_reward/std": 0.5572252869606018,
116
+ "rewards/code_format_reward/mean": 0.0,
117
+ "rewards/code_format_reward/std": 0.0,
118
+ "step": 4
119
+ },
120
+ {
121
+ "clip_ratio/high_max": 0.0,
122
+ "clip_ratio/high_mean": 0.0,
123
+ "clip_ratio/low_mean": 0.0,
124
+ "clip_ratio/low_min": 0.0,
125
+ "clip_ratio/region_mean": 0.0,
126
+ "completions/clipped_ratio": -0.25,
127
+ "completions/max_length": 8192.0,
128
+ "completions/max_terminated_length": 6331.0,
129
+ "completions/mean_length": 6324.3125,
130
+ "completions/mean_terminated_length": 2215.400146484375,
131
+ "completions/min_length": 905.0,
132
+ "completions/min_terminated_length": 905.0,
133
+ "epoch": 0.625,
134
+ "frac_reward_zero_std": 0.5,
135
+ "grad_norm": 0.24502748244021016,
136
+ "learning_rate": 1e-06,
137
+ "loss": -0.0402,
138
+ "num_tokens": 118167.0,
139
+ "reward": -0.6746212244033813,
140
+ "reward_std": 0.428866446018219,
141
+ "rewards/cf_code_reward/mean": -0.6746212244033813,
142
+ "rewards/cf_code_reward/std": 0.7365784049034119,
143
+ "rewards/code_format_reward/mean": 0.0,
144
+ "rewards/code_format_reward/std": 0.0,
145
+ "step": 5
146
+ },
147
+ {
148
+ "clip_ratio/high_max": 0.0,
149
+ "clip_ratio/high_mean": 0.0,
150
+ "clip_ratio/low_mean": 0.0,
151
+ "clip_ratio/low_min": 0.0,
152
+ "clip_ratio/region_mean": 0.0,
153
+ "completions/clipped_ratio": -1.25,
154
+ "completions/max_length": 8192.0,
155
+ "completions/max_terminated_length": 5359.0,
156
+ "completions/mean_length": 5666.6875,
157
+ "completions/mean_terminated_length": 3702.5556640625,
158
+ "completions/min_length": 1851.0,
159
+ "completions/min_terminated_length": 1851.0,
160
+ "epoch": 0.75,
161
+ "frac_reward_zero_std": 0.25,
162
+ "grad_norm": 0.21413429634446854,
163
+ "learning_rate": 1e-06,
164
+ "loss": -0.0416,
165
+ "num_tokens": 161958.0,
166
+ "reward": -0.5037499666213989,
167
+ "reward_std": 0.39220669865608215,
168
+ "rewards/cf_code_reward/mean": -0.5037499666213989,
169
+ "rewards/cf_code_reward/std": 0.6423693299293518,
170
+ "rewards/code_format_reward/mean": 0.0,
171
+ "rewards/code_format_reward/std": 0.0,
172
+ "step": 6
173
+ },
174
+ {
175
+ "clip_ratio/high_max": 0.0,
176
+ "clip_ratio/high_mean": 0.0,
177
+ "clip_ratio/low_mean": 0.0,
178
+ "clip_ratio/low_min": 0.0,
179
+ "clip_ratio/region_mean": 0.0,
180
+ "completions/clipped_ratio": 0.75,
181
+ "completions/max_length": 8192.0,
182
+ "completions/max_terminated_length": 7498.0,
183
+ "completions/mean_length": 8148.625,
184
+ "completions/mean_terminated_length": 7498.0,
185
+ "completions/min_length": 7498.0,
186
+ "completions/min_terminated_length": 7498.0,
187
+ "epoch": 0.875,
188
+ "frac_reward_zero_std": 0.75,
189
+ "grad_norm": 0.15542675545894089,
190
+ "learning_rate": 1e-06,
191
+ "loss": -0.0386,
192
+ "num_tokens": 182668.0,
193
+ "reward": -0.9437500238418579,
194
+ "reward_std": 0.11249999701976776,
195
+ "rewards/cf_code_reward/mean": -0.9437500238418579,
196
+ "rewards/cf_code_reward/std": 0.22499999403953552,
197
+ "rewards/code_format_reward/mean": 0.0,
198
+ "rewards/code_format_reward/std": 0.0,
199
+ "step": 7
200
+ },
201
+ {
202
+ "clip_ratio/high_max": 0.0,
203
+ "clip_ratio/high_mean": 0.0,
204
+ "clip_ratio/low_mean": 0.0,
205
+ "clip_ratio/low_min": 0.0,
206
+ "clip_ratio/region_mean": 0.0,
207
+ "completions/clipped_ratio": 0.25,
208
+ "completions/max_length": 8192.0,
209
+ "completions/max_terminated_length": 7341.0,
210
+ "completions/mean_length": 7525.6875,
211
+ "completions/mean_terminated_length": 4638.33349609375,
212
+ "completions/min_length": 2906.0,
213
+ "completions/min_terminated_length": 2906.0,
214
+ "epoch": 1.0,
215
+ "frac_reward_zero_std": 0.5,
216
+ "grad_norm": 0.17998151407614052,
217
+ "learning_rate": 1e-06,
218
+ "loss": -0.0646,
219
+ "num_tokens": 209383.0,
220
+ "reward": -0.8104166984558105,
221
+ "reward_std": 0.2664600610733032,
222
+ "rewards/cf_code_reward/mean": -0.8104166984558105,
223
+ "rewards/cf_code_reward/std": 0.4091035723686218,
224
+ "rewards/code_format_reward/mean": 0.0,
225
+ "rewards/code_format_reward/std": 0.0,
226
+ "step": 8
227
+ },
228
+ {
229
+ "epoch": 1.0,
230
+ "step": 8,
231
+ "total_flos": 0.0,
232
+ "train_loss": -0.039116610772907734,
233
+ "train_runtime": 615.0598,
234
+ "train_samples_per_second": 0.052,
235
+ "train_steps_per_second": 0.013
236
+ }
237
+ ],
238
+ "logging_steps": 1,
239
+ "max_steps": 8,
240
+ "num_input_tokens_seen": 209383,
241
+ "num_train_epochs": 1,
242
+ "save_steps": 1,
243
+ "stateful_callbacks": {
244
+ "TrainerControl": {
245
+ "args": {
246
+ "should_epoch_stop": false,
247
+ "should_evaluate": false,
248
+ "should_log": false,
249
+ "should_save": true,
250
+ "should_training_stop": true
251
+ },
252
+ "attributes": {}
253
+ }
254
+ },
255
+ "total_flos": 0.0,
256
+ "train_batch_size": 4,
257
+ "trial_name": null,
258
+ "trial_params": null
259
+ }