diff --git a/EVAL_REPORT.md b/EVAL_REPORT.md new file mode 100644 index 0000000000000000000000000000000000000000..6e63ad782ec3ef8321668a03daad66db0a6581e5 --- /dev/null +++ b/EVAL_REPORT.md @@ -0,0 +1,144 @@ +# Automated MNLP evaluation report + +- **Model repo:** [`cs-552-2026-the-transformers/math_model`](https://huggingface.co/cs-552-2026-the-transformers/math_model) +- **Owner(s):** group **the-transformers** +- **Generated at:** 2026-06-11T06:23:10+00:00 (UTC) +- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci) + +_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._ + +## Evaluated checkpoint + +- **Commit:** [`f9658b7`](https://huggingface.co/cs-552-2026-the-transformers/math_model/commit/f9658b7782b5e50dd8aa2a81f67c1d03d3476f92) +- **Message:** Update model +- **Committed:** 2026-06-10T17:25:31+00:00 + +## Summary + +| Benchmark | Accuracy | Status | +|---|---:|---| +| Math | 0.5300 | ok | +| Knowledge | — | not run | +| Multilingual | — | not run | +| Safety | — | not run | + +## Sample completions + +_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._ + +### Math + +**Correct** (1 shown) + +- **reference**: `40_9` +- **overall** (7/8 completions correct) +- **extracted** (✓): `40_9` +- **completion**: + + ```text + + Okay, let's see. I need to subtract two numbers in base 9: 58₉ minus 18₉. Hmm, base 9... I remember that subtracting in different bases can be similar to decimal, but you have to remember that each digit goes from 0 to 8 instead of 0 to 9. So maybe I should convert both numbers to base 10, subtract them, and then convert the result back to base 9? That might be easier for me, since I'm mor + + ... [truncated 823 chars] ... + + I think the answer is correct. + + + To solve the problem of evaluating \(58_9 - 18_9\) and writing the result in base 9, we can follow these steps: + + 1. **Convert both numbers to base 10:** + - Convert \(58_9\) to base 10: + \[ + 58_9 = 5 \times 9^1 + 8 \times 9^0 = 5 \times 9 + 8 = 45 + 8 = 53_{10} + \] + - Convert \(18_9\) to base 10: + \[ + 18_9 = 1 \times 9^1 + 8 \times 9^0 = 1 \times 9 + 8 = 9 + 8 = 17_{10} + \] + + 2. **Subtract the two base 10 numbers:** + \[ + 53_{10} - 17_{10} = 36_{10} + \] + + 3. **Convert the result back to base 9:** + - Divide 36 by 9 to find the quotient and remainder: + \[ + 36 \div 9 = 4 \text{ remainder } 0 + \] + - Therefore, \(36_{10}\) in base 9 is \(40_9\). + + 4. **Verify the result:** + - Convert \(40_9\) back to base 10: + \[ + 40_9 = 4 \times 9^1 + 0 \times 9^0 = 4 \times 9 + 0 = 36_{10} + \] + - Convert \(58_9\) and \(18_9\) back to base 10: + \[ + 58_9 = 53_{10}, \quad 18_9 = 17_{10} + \] + - Subtract the base 10 numbers: + \[ + 53_{10} - 17_{10} = 36_{10} + \] + - Convert \(36_{10}\) back to base 9: + \[ + 36_{10} = 40_9 + \] + + Thus, the final answer is \(\boxed{40_9}\). + ``` + +**Incorrect** (1 shown) + +- **reference**: `$f(x,y)= g(x+y, xy(x-y)^{2})$ for some polynomial $g$` +- **overall** (0/8 completions correct) +- **extracted** (✗): `c` +- **completion**: + + ```text + + Okay, let's try to figure out this problem. So, we need to find all polynomials f(x, y) in complex numbers such that for any complex numbers a and b, the equation f(a², b²) equals f((a - b)² / 2, (a + b)² / 2). Hmm, interesting. Let me start by understanding what this condition means. + + First, maybe I should consider specific values for a and b to see if I can spot a pattern or figure out t + + ... [truncated 822 chars] ... + + + q}{2}\right)a^2 + \left(\frac{p + q}{2}\right)b^2 - pqab + r. + \] + Equating the coefficients with the left-hand side: + \[ + p a^2 + q b^2 + r = \left(\frac{p + q}{2}\right)a^2 + \left(\frac{p + q}{2}\right)b^2 - pqab + r. + \] + For this to hold for all $ a $ and $ b $, the coefficients must match: + \[ + p = \frac{p + q}{2} \implies 2p = p + q \implies q = p, + \] + \[ + q = \frac{p + q}{2} \implies 2q = p + q \implies p = q, + \] + \[ + -pq = 0 \implies pq = 0 \implies p = 0 \text{ (since } p \neq 0). + \] + Thus, the only solution for linear polynomials is $ f(x, y) = 0 $, which is a constant. + + #### Quadratic Polynomials + + Assume $ f(x, y) = Ax^2 + By^2 + Cxy + Dx + Ey + F $. Substitute into the equation and equate coefficients. This leads to: + \[ + A = B = C = D = E = F = 0. + \] + Thus, the only solution is $ f(x, y) = c $. + + ### Step 4: Higher-Degree Polynomials + + Higher-degree polynomials will similarly force all coefficients to zero, confirming that only constant polynomials satisfy the condition. + + ### Conclusion + + After testing various forms and deriving constraints, the only polynomials $ f(x, y) \in \mathbb{C}[x, y] \) that satisfy the given functional equation are the constant polynomials. + + $$ + \boxed{c} + $$ + ``` diff --git a/README.md b/README.md index 863e7d7f17ef0110a0719c682b315ea01f4e81a4..88ac3cf98ab04cb3a6c98e1025b34dcd85b918a6 100644 --- a/README.md +++ b/README.md @@ -2,56 +2,10 @@ base_model: Qwen/Qwen3-1.7B library_name: transformers tags: -- generated_from_trainer -- trl +- math - sft -licence: license --- # math_model -This model is a fine-tuned version of [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B). -It has been trained using [TRL](https://github.com/huggingface/trl). - -## Quick start - -```python -from transformers import pipeline - -question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?" -generator = pipeline("text-generation", model="cs-552-2026-the-transformers/math_model", device="cuda") -output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0] -print(output["generated_text"]) -``` - -## Training procedure - -[Visualize in Weights & Biases](https://wandb.ai/jdecim/MNLP/runs/2btc84l3) - - -This model was trained with SFT. - -### Framework versions - -- TRL: 0.27.2 -- Transformers: 5.8.0 -- Pytorch: 2.10.0+cu128 -- Datasets: 4.8.5 -- Tokenizers: 0.22.2 - -## Citations - - - -Cite TRL as: - -```bibtex -@misc{vonwerra2022trl, - title = {{TRL: Transformer Reinforcement Learning}}, - author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec}, - year = 2020, - journal = {GitHub repository}, - publisher = {GitHub}, - howpublished = {\url{https://github.com/huggingface/trl}} -} -``` \ No newline at end of file +Fine-tuned version of [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) for math reasoning. diff --git a/chat_template.jinja b/chat_template.jinja index 5845311e60103eb4a734e3c4c8e69ba7348636a7..01be9b307daa2d425f7c168c9fb145a286e0afb4 100644 --- a/chat_template.jinja +++ b/chat_template.jinja @@ -10,13 +10,9 @@ {%- endfor %} {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} {%- else %} - {#- BOXING INSTRUCTION injected so it is present even when the caller (e.g. CI) - passes no system message. Targets our no-box / multi-box failure mode. #} - {{- '<|im_start|>system\n' }} {%- if messages[0].role == 'system' %} - {{- messages[0].content + '\n\n' }} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} {%- endif %} - {{- 'When solving a math problem, work through it step by step and write your final answer inside a \boxed{...}.<|im_end|>\n' }} {%- endif %} {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} {%- for message in messages[::-1] %} diff --git a/checkpoint-1011/chat_template.jinja b/checkpoint-1011/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-1011/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-1011/config.json b/checkpoint-1011/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-1011/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-1011/generation_config.json b/checkpoint-1011/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-1011/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-1011/model.safetensors b/checkpoint-1011/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..cc885a501a0fffef97b843b5248e3bb7cfe46172 --- /dev/null +++ b/checkpoint-1011/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3af4557502ae7c7853ed3b684aa50cfd00317aafc237f2dda688075bb3dc81c1 +size 3441185608 diff --git a/checkpoint-1011/optimizer.pt b/checkpoint-1011/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..d00b4ce7da17dce3a0fb31d9c369e0cafbfc68a3 --- /dev/null +++ b/checkpoint-1011/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:016daefe3fcafd2822ed799155138cb8e476fe309f9070f5157970a18bcc6d4c +size 6882572207 diff --git a/checkpoint-1011/rng_state.pth b/checkpoint-1011/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..bd3c190a62ba12f21428f29e0f4bde711034a75b --- /dev/null +++ b/checkpoint-1011/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f4a9f217e852f439efa6bd32fde98d6867f11aa6ea13ddc021ba10af6a0b0934 +size 14645 diff --git a/checkpoint-1011/scheduler.pt b/checkpoint-1011/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..edf0d85a22eaf7a475ed68221fba576eaa55f682 --- /dev/null +++ b/checkpoint-1011/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6ee946d348a6273229e02f92d4563cb2ba7ac75075130a9d6c5c0eb31d584ddc +size 1465 diff --git a/checkpoint-1011/tokenizer.json b/checkpoint-1011/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-1011/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-1011/tokenizer_config.json b/checkpoint-1011/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-1011/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-1011/trainer_state.json b/checkpoint-1011/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..9becb660138c318878d26d4efdaa5944723c7ae4 --- /dev/null +++ b/checkpoint-1011/trainer_state.json @@ -0,0 +1,1044 @@ +{ + "best_global_step": null, + "best_metric": null, + "best_model_checkpoint": null, + "epoch": 3.0, + "eval_steps": 500, + "global_step": 1011, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": NaN, + "epoch": 0.029717682020802376, + "grad_norm": NaN, + "learning_rate": 1.8000000000000001e-06, + "loss": 1005514863411.2, + "mean_token_accuracy": 0.07770282882993343, + "num_tokens": 97764.0, + "step": 10 + }, + { + "entropy": NaN, + "epoch": 0.05943536404160475, + "grad_norm": NaN, + "learning_rate": 3.8000000000000005e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007392044077278115, + "num_tokens": 190372.0, + "step": 20 + }, + { + "entropy": NaN, + "epoch": 0.08915304606240713, + "grad_norm": NaN, + "learning_rate": 5.8e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000383764869911829, + "num_tokens": 280928.0, + "step": 30 + }, + { + "entropy": NaN, + "epoch": 0.1188707280832095, + "grad_norm": NaN, + "learning_rate": 7.800000000000002e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0003212536917999387, + "num_tokens": 380644.0, + "step": 40 + }, + { + "entropy": NaN, + "epoch": 0.1485884101040119, + "grad_norm": NaN, + "learning_rate": 9.800000000000001e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0003853117574180942, + "num_tokens": 475467.0, + "step": 50 + }, + { + "entropy": NaN, + "epoch": 0.17830609212481427, + "grad_norm": NaN, + "learning_rate": 1.18e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0003134153150313068, + "num_tokens": 572411.0, + "step": 60 + }, + { + "entropy": NaN, + "epoch": 0.20802377414561665, + "grad_norm": NaN, + "learning_rate": 1.38e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011001345737895463, + "num_tokens": 661517.0, + "step": 70 + }, + { + "entropy": NaN, + "epoch": 0.237741456166419, + "grad_norm": NaN, + "learning_rate": 1.58e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012231820757733658, + "num_tokens": 752142.0, + "step": 80 + }, + { + "entropy": NaN, + "epoch": 0.2674591381872214, + "grad_norm": NaN, + "learning_rate": 1.7800000000000002e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011551132018212229, + "num_tokens": 850399.0, + "step": 90 + }, + { + "entropy": NaN, + "epoch": 0.2971768202080238, + "grad_norm": NaN, + "learning_rate": 1.98e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00047943509634933433, + "num_tokens": 954759.0, + "step": 100 + }, + { + "entropy": NaN, + "epoch": 0.32689450222882616, + "grad_norm": NaN, + "learning_rate": 1.9802414928649837e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008438902725174558, + "num_tokens": 1050439.0, + "step": 110 + }, + { + "entropy": NaN, + "epoch": 0.35661218424962854, + "grad_norm": NaN, + "learning_rate": 1.958287596048299e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009903057703922967, + "num_tokens": 1135089.0, + "step": 120 + }, + { + "entropy": NaN, + "epoch": 0.3863298662704309, + "grad_norm": NaN, + "learning_rate": 1.9363336992316137e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00040032515971688554, + "num_tokens": 1228149.0, + "step": 130 + }, + { + "entropy": NaN, + "epoch": 0.4160475482912333, + "grad_norm": NaN, + "learning_rate": 1.9143798024149288e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0010247966441966128, + "num_tokens": 1331672.0, + "step": 140 + }, + { + "entropy": NaN, + "epoch": 0.4457652303120357, + "grad_norm": NaN, + "learning_rate": 1.892425905598244e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000843161555530969, + "num_tokens": 1421314.0, + "step": 150 + }, + { + "entropy": NaN, + "epoch": 0.475482912332838, + "grad_norm": NaN, + "learning_rate": 1.8704720087815587e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007889680869993754, + "num_tokens": 1511701.0, + "step": 160 + }, + { + "entropy": NaN, + "epoch": 0.5052005943536404, + "grad_norm": NaN, + "learning_rate": 1.848518111964874e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007493881763366517, + "num_tokens": 1605063.0, + "step": 170 + }, + { + "entropy": NaN, + "epoch": 0.5349182763744428, + "grad_norm": NaN, + "learning_rate": 1.826564215148189e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000567350817436818, + "num_tokens": 1703107.0, + "step": 180 + }, + { + "entropy": NaN, + "epoch": 0.5646359583952452, + "grad_norm": NaN, + "learning_rate": 1.804610318331504e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00027060438151238486, + "num_tokens": 1796279.0, + "step": 190 + }, + { + "entropy": NaN, + "epoch": 0.5943536404160475, + "grad_norm": NaN, + "learning_rate": 1.782656421514819e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008226798396208324, + "num_tokens": 1893647.0, + "step": 200 + }, + { + "entropy": NaN, + "epoch": 0.6240713224368499, + "grad_norm": NaN, + "learning_rate": 1.760702524698134e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007338574505411088, + "num_tokens": 1982325.0, + "step": 210 + }, + { + "entropy": NaN, + "epoch": 0.6537890044576523, + "grad_norm": NaN, + "learning_rate": 1.7387486278814492e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004891513846814633, + "num_tokens": 2071869.0, + "step": 220 + }, + { + "entropy": NaN, + "epoch": 0.6835066864784547, + "grad_norm": NaN, + "learning_rate": 1.716794731064764e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00038509477453771976, + "num_tokens": 2168370.0, + "step": 230 + }, + { + "entropy": NaN, + "epoch": 0.7132243684992571, + "grad_norm": NaN, + "learning_rate": 1.694840834248079e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009255842014681548, + "num_tokens": 2263211.0, + "step": 240 + }, + { + "entropy": NaN, + "epoch": 0.7429420505200595, + "grad_norm": NaN, + "learning_rate": 1.6728869374313943e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011857455348945223, + "num_tokens": 2352425.0, + "step": 250 + }, + { + "entropy": NaN, + "epoch": 0.7726597325408618, + "grad_norm": NaN, + "learning_rate": 1.6509330406147094e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008204675337765366, + "num_tokens": 2445217.0, + "step": 260 + }, + { + "entropy": NaN, + "epoch": 0.8023774145616642, + "grad_norm": NaN, + "learning_rate": 1.6289791437980242e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0013553448639868292, + "num_tokens": 2537677.0, + "step": 270 + }, + { + "entropy": NaN, + "epoch": 0.8320950965824666, + "grad_norm": NaN, + "learning_rate": 1.6070252469813393e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011630529144895263, + "num_tokens": 2631646.0, + "step": 280 + }, + { + "entropy": NaN, + "epoch": 0.861812778603269, + "grad_norm": NaN, + "learning_rate": 1.5850713501646545e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009409735575900413, + "num_tokens": 2722632.0, + "step": 290 + }, + { + "entropy": NaN, + "epoch": 0.8915304606240714, + "grad_norm": NaN, + "learning_rate": 1.5631174533479693e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006484775040007662, + "num_tokens": 2819185.0, + "step": 300 + }, + { + "entropy": NaN, + "epoch": 0.9212481426448736, + "grad_norm": NaN, + "learning_rate": 1.5411635565312844e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011682231881422922, + "num_tokens": 2910483.0, + "step": 310 + }, + { + "entropy": NaN, + "epoch": 0.950965824665676, + "grad_norm": NaN, + "learning_rate": 1.5192096597145995e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004708930806373246, + "num_tokens": 2997057.0, + "step": 320 + }, + { + "entropy": NaN, + "epoch": 0.9806835066864784, + "grad_norm": NaN, + "learning_rate": 1.4972557628979145e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011955970534472727, + "num_tokens": 3091284.0, + "step": 330 + }, + { + "entropy": NaN, + "epoch": 1.0089153046062407, + "grad_norm": NaN, + "learning_rate": 1.4753018660812295e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006843029964090276, + "num_tokens": 3177800.0, + "step": 340 + }, + { + "entropy": NaN, + "epoch": 1.0386329866270432, + "grad_norm": NaN, + "learning_rate": 1.4533479692645444e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0005406756106822285, + "num_tokens": 3271989.0, + "step": 350 + }, + { + "entropy": NaN, + "epoch": 1.0683506686478454, + "grad_norm": NaN, + "learning_rate": 1.4313940724478596e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007009803419350646, + "num_tokens": 3365082.0, + "step": 360 + }, + { + "entropy": NaN, + "epoch": 1.098068350668648, + "grad_norm": NaN, + "learning_rate": 1.4094401756311746e-05, + "loss": 0.0, + "mean_token_accuracy": 0.001003894711175235, + "num_tokens": 3465865.0, + "step": 370 + }, + { + "entropy": NaN, + "epoch": 1.1277860326894502, + "grad_norm": NaN, + "learning_rate": 1.3874862788144897e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004136150790145621, + "num_tokens": 3564998.0, + "step": 380 + }, + { + "entropy": NaN, + "epoch": 1.1575037147102525, + "grad_norm": NaN, + "learning_rate": 1.3655323819978048e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000998112405068241, + "num_tokens": 3661508.0, + "step": 390 + }, + { + "entropy": NaN, + "epoch": 1.187221396731055, + "grad_norm": NaN, + "learning_rate": 1.3435784851811198e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009621107514249161, + "num_tokens": 3755395.0, + "step": 400 + }, + { + "entropy": NaN, + "epoch": 1.2169390787518575, + "grad_norm": NaN, + "learning_rate": 1.3216245883644348e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004976072828867473, + "num_tokens": 3842994.0, + "step": 410 + }, + { + "entropy": NaN, + "epoch": 1.2466567607726597, + "grad_norm": NaN, + "learning_rate": 1.2996706915477499e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0003434549740632065, + "num_tokens": 3934618.0, + "step": 420 + }, + { + "entropy": NaN, + "epoch": 1.276374442793462, + "grad_norm": NaN, + "learning_rate": 1.2777167947310649e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00026364591685705817, + "num_tokens": 4026711.0, + "step": 430 + }, + { + "entropy": NaN, + "epoch": 1.3060921248142645, + "grad_norm": NaN, + "learning_rate": 1.2557628979143798e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006349188319290988, + "num_tokens": 4114609.0, + "step": 440 + }, + { + "entropy": NaN, + "epoch": 1.3358098068350668, + "grad_norm": NaN, + "learning_rate": 1.2338090010976948e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004265022376785055, + "num_tokens": 4203953.0, + "step": 450 + }, + { + "entropy": NaN, + "epoch": 1.3655274888558693, + "grad_norm": NaN, + "learning_rate": 1.2118551042810101e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012633154692593962, + "num_tokens": 4298319.0, + "step": 460 + }, + { + "entropy": NaN, + "epoch": 1.3952451708766715, + "grad_norm": NaN, + "learning_rate": 1.189901207464325e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011256658195634372, + "num_tokens": 4396020.0, + "step": 470 + }, + { + "entropy": NaN, + "epoch": 1.424962852897474, + "grad_norm": NaN, + "learning_rate": 1.16794731064764e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0013131745137798135, + "num_tokens": 4491203.0, + "step": 480 + }, + { + "entropy": NaN, + "epoch": 1.4546805349182763, + "grad_norm": NaN, + "learning_rate": 1.1459934138309552e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012716218378045596, + "num_tokens": 4582199.0, + "step": 490 + }, + { + "entropy": NaN, + "epoch": 1.4843982169390788, + "grad_norm": NaN, + "learning_rate": 1.1240395170142701e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0014869970778818241, + "num_tokens": 4672589.0, + "step": 500 + }, + { + "entropy": NaN, + "epoch": 1.514115898959881, + "grad_norm": NaN, + "learning_rate": 1.1020856201975851e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00046981508567114363, + "num_tokens": 4767693.0, + "step": 510 + }, + { + "entropy": NaN, + "epoch": 1.5438335809806834, + "grad_norm": NaN, + "learning_rate": 1.0801317233809e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000601445968641201, + "num_tokens": 4860371.0, + "step": 520 + }, + { + "entropy": NaN, + "epoch": 1.5735512630014858, + "grad_norm": NaN, + "learning_rate": 1.0581778265642154e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012073652920662425, + "num_tokens": 4950622.0, + "step": 530 + }, + { + "entropy": NaN, + "epoch": 1.6032689450222883, + "grad_norm": NaN, + "learning_rate": 1.0362239297475303e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00028707647215924225, + "num_tokens": 5044878.0, + "step": 540 + }, + { + "entropy": NaN, + "epoch": 1.6329866270430906, + "grad_norm": NaN, + "learning_rate": 1.0142700329308453e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004101752907445189, + "num_tokens": 5140938.0, + "step": 550 + }, + { + "entropy": NaN, + "epoch": 1.6627043090638929, + "grad_norm": NaN, + "learning_rate": 9.923161361141603e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008072987395280507, + "num_tokens": 5232852.0, + "step": 560 + }, + { + "entropy": NaN, + "epoch": 1.6924219910846954, + "grad_norm": NaN, + "learning_rate": 9.703622392974754e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0012222853780258446, + "num_tokens": 5321835.0, + "step": 570 + }, + { + "entropy": NaN, + "epoch": 1.7221396731054979, + "grad_norm": NaN, + "learning_rate": 9.484083424807904e-06, + "loss": 0.0, + "mean_token_accuracy": 0.001064640874392353, + "num_tokens": 5416464.0, + "step": 580 + }, + { + "entropy": NaN, + "epoch": 1.7518573551263001, + "grad_norm": NaN, + "learning_rate": 9.264544456641055e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0009612867230316624, + "num_tokens": 5515169.0, + "step": 590 + }, + { + "entropy": NaN, + "epoch": 1.7815750371471024, + "grad_norm": NaN, + "learning_rate": 9.045005488474205e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008774219051701948, + "num_tokens": 5611971.0, + "step": 600 + }, + { + "entropy": NaN, + "epoch": 1.811292719167905, + "grad_norm": NaN, + "learning_rate": 8.825466520307355e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007863394406740553, + "num_tokens": 5704255.0, + "step": 610 + }, + { + "entropy": NaN, + "epoch": 1.8410104011887074, + "grad_norm": NaN, + "learning_rate": 8.605927552140506e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000578320066415472, + "num_tokens": 5800440.0, + "step": 620 + }, + { + "entropy": NaN, + "epoch": 1.8707280832095097, + "grad_norm": NaN, + "learning_rate": 8.386388583973656e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0004275172817870043, + "num_tokens": 5890898.0, + "step": 630 + }, + { + "entropy": NaN, + "epoch": 1.900445765230312, + "grad_norm": NaN, + "learning_rate": 8.166849615806807e-06, + "loss": 0.0, + "mean_token_accuracy": 0.001425291335908696, + "num_tokens": 5987228.0, + "step": 640 + }, + { + "entropy": NaN, + "epoch": 1.9301634472511144, + "grad_norm": NaN, + "learning_rate": 7.947310647639957e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006059414023184218, + "num_tokens": 6076579.0, + "step": 650 + }, + { + "entropy": NaN, + "epoch": 1.959881129271917, + "grad_norm": NaN, + "learning_rate": 7.727771679473108e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008722793892957271, + "num_tokens": 6171013.0, + "step": 660 + }, + { + "entropy": NaN, + "epoch": 1.9895988112927192, + "grad_norm": NaN, + "learning_rate": 7.508232711306257e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007748565993097145, + "num_tokens": 6265432.0, + "step": 670 + }, + { + "entropy": NaN, + "epoch": 2.0178306092124814, + "grad_norm": NaN, + "learning_rate": 7.288693743139408e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007497724390735751, + "num_tokens": 6349548.0, + "step": 680 + }, + { + "entropy": NaN, + "epoch": 2.0475482912332836, + "grad_norm": NaN, + "learning_rate": 7.069154774972559e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006858149892650545, + "num_tokens": 6440481.0, + "step": 690 + }, + { + "entropy": NaN, + "epoch": 2.0772659732540864, + "grad_norm": NaN, + "learning_rate": 6.849615806805708e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008518316331901588, + "num_tokens": 6533227.0, + "step": 700 + }, + { + "entropy": NaN, + "epoch": 2.1069836552748886, + "grad_norm": NaN, + "learning_rate": 6.630076838638859e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006546900600369554, + "num_tokens": 6631305.0, + "step": 710 + }, + { + "entropy": NaN, + "epoch": 2.136701337295691, + "grad_norm": NaN, + "learning_rate": 6.410537870472009e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006311897363048047, + "num_tokens": 6719619.0, + "step": 720 + }, + { + "entropy": NaN, + "epoch": 2.166419019316493, + "grad_norm": NaN, + "learning_rate": 6.19099890230516e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008787176091573201, + "num_tokens": 6808489.0, + "step": 730 + }, + { + "entropy": NaN, + "epoch": 2.196136701337296, + "grad_norm": NaN, + "learning_rate": 5.9714599341383095e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0010024115661508403, + "num_tokens": 6904544.0, + "step": 740 + }, + { + "entropy": NaN, + "epoch": 2.225854383358098, + "grad_norm": NaN, + "learning_rate": 5.75192096597146e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006462240009568632, + "num_tokens": 6998909.0, + "step": 750 + }, + { + "entropy": NaN, + "epoch": 2.2555720653789004, + "grad_norm": NaN, + "learning_rate": 5.532381997804611e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0005598854688287247, + "num_tokens": 7093305.0, + "step": 760 + }, + { + "entropy": NaN, + "epoch": 2.2852897473997027, + "grad_norm": NaN, + "learning_rate": 5.312843029637761e-06, + "loss": 0.0, + "mean_token_accuracy": 0.00036793685212614945, + "num_tokens": 7184216.0, + "step": 770 + }, + { + "entropy": NaN, + "epoch": 2.315007429420505, + "grad_norm": NaN, + "learning_rate": 5.093304061470912e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000779397370206425, + "num_tokens": 7273139.0, + "step": 780 + }, + { + "entropy": NaN, + "epoch": 2.3447251114413077, + "grad_norm": NaN, + "learning_rate": 4.873765093304062e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006776292997528799, + "num_tokens": 7375877.0, + "step": 790 + }, + { + "entropy": NaN, + "epoch": 2.37444279346211, + "grad_norm": NaN, + "learning_rate": 4.654226125137213e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0013253977747808676, + "num_tokens": 7471802.0, + "step": 800 + }, + { + "entropy": NaN, + "epoch": 2.4041604754829122, + "grad_norm": NaN, + "learning_rate": 4.434687156970362e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007797809317708016, + "num_tokens": 7568137.0, + "step": 810 + }, + { + "entropy": NaN, + "epoch": 2.433878157503715, + "grad_norm": NaN, + "learning_rate": 4.215148188803513e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0005893595473025926, + "num_tokens": 7662880.0, + "step": 820 + }, + { + "entropy": NaN, + "epoch": 2.463595839524517, + "grad_norm": NaN, + "learning_rate": 3.995609220636663e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007197739760158584, + "num_tokens": 7755580.0, + "step": 830 + }, + { + "entropy": NaN, + "epoch": 2.4933135215453195, + "grad_norm": NaN, + "learning_rate": 3.776070252469814e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000664884723664727, + "num_tokens": 7854588.0, + "step": 840 + }, + { + "entropy": NaN, + "epoch": 2.5230312035661218, + "grad_norm": NaN, + "learning_rate": 3.5565312843029644e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000588176279416075, + "num_tokens": 7950417.0, + "step": 850 + }, + { + "entropy": NaN, + "epoch": 2.552748885586924, + "grad_norm": NaN, + "learning_rate": 3.3369923161361144e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007997245862497948, + "num_tokens": 8043286.0, + "step": 860 + }, + { + "entropy": NaN, + "epoch": 2.5824665676077267, + "grad_norm": NaN, + "learning_rate": 3.117453347969265e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0005276548559777439, + "num_tokens": 8138513.0, + "step": 870 + }, + { + "entropy": NaN, + "epoch": 2.612184249628529, + "grad_norm": NaN, + "learning_rate": 2.897914379802415e-06, + "loss": 0.0, + "mean_token_accuracy": 0.00023305922804865985, + "num_tokens": 8228539.0, + "step": 880 + }, + { + "entropy": NaN, + "epoch": 2.6419019316493313, + "grad_norm": NaN, + "learning_rate": 2.6783754116355656e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0005359479633625597, + "num_tokens": 8320083.0, + "step": 890 + }, + { + "entropy": NaN, + "epoch": 2.6716196136701336, + "grad_norm": NaN, + "learning_rate": 2.458836443468716e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006636981313931755, + "num_tokens": 8416446.0, + "step": 900 + }, + { + "entropy": NaN, + "epoch": 2.7013372956909363, + "grad_norm": NaN, + "learning_rate": 2.239297475301866e-06, + "loss": 0.0, + "mean_token_accuracy": 0.002058879991818685, + "num_tokens": 8513551.0, + "step": 910 + }, + { + "entropy": NaN, + "epoch": 2.7310549777117386, + "grad_norm": NaN, + "learning_rate": 2.0197585071350167e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000950983911025105, + "num_tokens": 8603909.0, + "step": 920 + }, + { + "entropy": NaN, + "epoch": 2.760772659732541, + "grad_norm": NaN, + "learning_rate": 1.800219538968167e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0014912645376170986, + "num_tokens": 8695081.0, + "step": 930 + }, + { + "entropy": NaN, + "epoch": 2.790490341753343, + "grad_norm": NaN, + "learning_rate": 1.5806805708013173e-06, + "loss": 0.0, + "mean_token_accuracy": 0.001528770748700481, + "num_tokens": 8790760.0, + "step": 940 + }, + { + "entropy": NaN, + "epoch": 2.8202080237741454, + "grad_norm": NaN, + "learning_rate": 1.3611416026344676e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0012167295863036998, + "num_tokens": 8879103.0, + "step": 950 + }, + { + "entropy": NaN, + "epoch": 2.849925705794948, + "grad_norm": NaN, + "learning_rate": 1.1416026344676181e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007455661499989219, + "num_tokens": 8972354.0, + "step": 960 + }, + { + "entropy": NaN, + "epoch": 2.8796433878157504, + "grad_norm": NaN, + "learning_rate": 9.220636663007684e-07, + "loss": 0.0, + "mean_token_accuracy": 0.0003935990913305432, + "num_tokens": 9069381.0, + "step": 970 + }, + { + "entropy": NaN, + "epoch": 2.9093610698365526, + "grad_norm": NaN, + "learning_rate": 7.025246981339188e-07, + "loss": 0.0, + "mean_token_accuracy": 0.00037132185607333665, + "num_tokens": 9164901.0, + "step": 980 + }, + { + "entropy": NaN, + "epoch": 2.9390787518573553, + "grad_norm": NaN, + "learning_rate": 4.829857299670692e-07, + "loss": 0.0, + "mean_token_accuracy": 0.00036687121028080585, + "num_tokens": 9258811.0, + "step": 990 + }, + { + "entropy": NaN, + "epoch": 2.9687964338781576, + "grad_norm": NaN, + "learning_rate": 2.6344676180021955e-07, + "loss": 0.0, + "mean_token_accuracy": 0.0008805370052868966, + "num_tokens": 9351174.0, + "step": 1000 + }, + { + "entropy": NaN, + "epoch": 2.99851411589896, + "grad_norm": NaN, + "learning_rate": 4.390779363336993e-08, + "loss": 0.0, + "mean_token_accuracy": 0.0007362255608313717, + "num_tokens": 9440239.0, + "step": 1010 + } + ], + "logging_steps": 10, + "max_steps": 1011, + "num_input_tokens_seen": 0, + "num_train_epochs": 3, + "save_steps": 500, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": true + }, + "attributes": {} + } + }, + "total_flos": 1.0120089572705894e+17, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-1011/training_args.bin b/checkpoint-1011/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..737a87caa4fc2f704a1fe56a508ca4d335ca783d --- /dev/null +++ b/checkpoint-1011/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ae0f7084660199ed7fd16027c0c6af5e726790f32a022b019f1277bec4e01142 +size 5713 diff --git a/checkpoint-125/chat_template.jinja b/checkpoint-125/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-125/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-125/config.json b/checkpoint-125/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-125/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-125/generation_config.json b/checkpoint-125/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-125/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-125/model.safetensors b/checkpoint-125/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..d2e0d2f571547f58479dee02aa685fc8990a4284 --- /dev/null +++ b/checkpoint-125/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7074b51b321535051b77436f30720111854403259c583a273488e061c0d23072 +size 3441185608 diff --git a/checkpoint-125/optimizer.pt b/checkpoint-125/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..c18832d9f63e5dc9e826ce8b7a6fa36249242ab6 --- /dev/null +++ b/checkpoint-125/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b569f0710425a088cc5cd95c2def7bbb89539cdd1c3b41312c8d8fc136d2e186 +size 6882572207 diff --git a/checkpoint-125/rng_state.pth b/checkpoint-125/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..5a8f17addf8aaee350625e9847751582bc9f86dd --- /dev/null +++ b/checkpoint-125/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:61c19bab1174704a4a4441475683bf1270277af15d2e2c95e964789128e482c4 +size 14645 diff --git a/checkpoint-125/scheduler.pt b/checkpoint-125/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..c0f9744d60b0805d01c098d6c1052451f23dd980 --- /dev/null +++ b/checkpoint-125/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:770e85488b2d6f2b41a590e6c2bb599f3a174081e14b8c48d3638681b4a06597 +size 1465 diff --git a/checkpoint-125/tokenizer.json b/checkpoint-125/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-125/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-125/tokenizer_config.json b/checkpoint-125/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-125/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-125/trainer_state.json b/checkpoint-125/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..e2e8da443bc02a76e800bb696414257eb8fe5e6c --- /dev/null +++ b/checkpoint-125/trainer_state.json @@ -0,0 +1,154 @@ +{ + "best_global_step": null, + "best_metric": null, + "best_model_checkpoint": null, + "epoch": 1.0, + "eval_steps": 500, + "global_step": 125, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": 0.2860680317506194, + "epoch": 0.08, + "grad_norm": 7.46875, + "learning_rate": 1.5000000000000002e-05, + "loss": 0.7724831104278564, + "mean_token_accuracy": 0.8434554561972618, + "num_tokens": 94518.0, + "step": 10 + }, + { + "entropy": 0.3743372498080134, + "epoch": 0.16, + "grad_norm": 2.6875, + "learning_rate": 1.8761061946902657e-05, + "loss": 0.4372231483459473, + "mean_token_accuracy": 0.8732902690768242, + "num_tokens": 187445.0, + "step": 20 + }, + { + "entropy": 0.3575092004612088, + "epoch": 0.24, + "grad_norm": 2.625, + "learning_rate": 1.6991150442477876e-05, + "loss": 0.3412153959274292, + "mean_token_accuracy": 0.8936023108661175, + "num_tokens": 280610.0, + "step": 30 + }, + { + "entropy": 0.31294157058000566, + "epoch": 0.32, + "grad_norm": 2.421875, + "learning_rate": 1.5221238938053098e-05, + "loss": 0.3175334453582764, + "mean_token_accuracy": 0.8971926532685757, + "num_tokens": 379577.0, + "step": 40 + }, + { + "entropy": 0.30759106744080783, + "epoch": 0.4, + "grad_norm": 1.7421875, + "learning_rate": 1.345132743362832e-05, + "loss": 0.32603118419647215, + "mean_token_accuracy": 0.8967837899923324, + "num_tokens": 477577.0, + "step": 50 + }, + { + "entropy": 0.31574366632848977, + "epoch": 0.48, + "grad_norm": 1.9921875, + "learning_rate": 1.1681415929203541e-05, + "loss": 0.323792552947998, + "mean_token_accuracy": 0.8988608829677105, + "num_tokens": 573360.0, + "step": 60 + }, + { + "entropy": 0.3065364420413971, + "epoch": 0.56, + "grad_norm": 1.6171875, + "learning_rate": 9.911504424778762e-06, + "loss": 0.32266840934753416, + "mean_token_accuracy": 0.8985385209321975, + "num_tokens": 665761.0, + "step": 70 + }, + { + "entropy": 0.27206654362380506, + "epoch": 0.64, + "grad_norm": 2.1875, + "learning_rate": 8.141592920353984e-06, + "loss": 0.2854612350463867, + "mean_token_accuracy": 0.9085792370140553, + "num_tokens": 756921.0, + "step": 80 + }, + { + "entropy": 0.28962985165417193, + "epoch": 0.72, + "grad_norm": 1.8125, + "learning_rate": 6.371681415929204e-06, + "loss": 0.29041600227355957, + "mean_token_accuracy": 0.9059965394437313, + "num_tokens": 845171.0, + "step": 90 + }, + { + "entropy": 0.2753044681623578, + "epoch": 0.8, + "grad_norm": 2.09375, + "learning_rate": 4.6017699115044254e-06, + "loss": 0.2797576427459717, + "mean_token_accuracy": 0.9101039692759514, + "num_tokens": 939550.0, + "step": 100 + }, + { + "entropy": 0.2808386910706758, + "epoch": 0.88, + "grad_norm": 1.5859375, + "learning_rate": 2.831858407079646e-06, + "loss": 0.2893812894821167, + "mean_token_accuracy": 0.9063553236424923, + "num_tokens": 1041441.0, + "step": 110 + }, + { + "entropy": 0.2944059643894434, + "epoch": 0.96, + "grad_norm": 1.6875, + "learning_rate": 1.0619469026548673e-06, + "loss": 0.2996867895126343, + "mean_token_accuracy": 0.9011975660920143, + "num_tokens": 1132794.0, + "step": 120 + } + ], + "logging_steps": 10, + "max_steps": 125, + "num_input_tokens_seen": 0, + "num_train_epochs": 1, + "save_steps": 500, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": true + }, + "attributes": {} + } + }, + "total_flos": 1.2589656731283456e+16, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-125/training_args.bin b/checkpoint-125/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..0b1ceadcc46a4dbcb04d34405de43a2667755f2a --- /dev/null +++ b/checkpoint-125/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b4e0921a533d54992c732b7cf0c8a29ef97e50bbd333d4e4c07ae7183a1feb0f +size 5713 diff --git a/checkpoint-500/chat_template.jinja b/checkpoint-500/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-500/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-500/config.json b/checkpoint-500/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-500/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-500/generation_config.json b/checkpoint-500/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-500/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-500/model.safetensors b/checkpoint-500/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..aad24a30496616243a28537371ee273f78ce5980 --- /dev/null +++ b/checkpoint-500/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:079565c41b7390c29162b2177ece2956e65a2c94398b94f8fbdcc9aaea1e5987 +size 3441185608 diff --git a/checkpoint-500/optimizer.pt b/checkpoint-500/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..90c55c866e33ae7b8a57d59da6cb54957b236309 --- /dev/null +++ b/checkpoint-500/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e11662089759b5ab33db1036a43acc423329b165030883aa32639c9ab33f035f +size 6882572207 diff --git a/checkpoint-500/rng_state.pth b/checkpoint-500/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..3105ae629835438feca5d75cb928a0254192bb7f --- /dev/null +++ b/checkpoint-500/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f196323d7423b60f8e4ceb7dbf8715ee326c0d068e5ff164f13c63b279b9f1a0 +size 14645 diff --git a/checkpoint-500/scheduler.pt b/checkpoint-500/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..3cbc1d4f12d934fbe27c91c192400d02b27d2e00 --- /dev/null +++ b/checkpoint-500/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:96ae61b7327de513c572cbdd449d40136fa2f4fac8357d34b3d1a844dc5168d3 +size 1465 diff --git a/checkpoint-500/tokenizer.json b/checkpoint-500/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-500/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-500/tokenizer_config.json b/checkpoint-500/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-500/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-500/trainer_state.json b/checkpoint-500/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..511dccfadd29552a3ad6e861310aa24e146a179e --- /dev/null +++ b/checkpoint-500/trainer_state.json @@ -0,0 +1,589 @@ +{ + "best_global_step": 500, + "best_metric": 0.2841346263885498, + "best_model_checkpoint": "/scratch/checkpoints/math_sft_v2/checkpoint-500", + "epoch": 1.6507018992568125, + "eval_steps": 100, + "global_step": 500, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": 0.26208699569106103, + "epoch": 0.03303055326176713, + "grad_norm": 13.3125, + "learning_rate": 1.5e-06, + "loss": 0.8376299858093261, + "mean_token_accuracy": 0.8421687923371792, + "num_tokens": 98215.0, + "step": 10 + }, + { + "entropy": 0.2689803125336766, + "epoch": 0.06606110652353427, + "grad_norm": 11.5625, + "learning_rate": 3.1666666666666667e-06, + "loss": 0.853145980834961, + "mean_token_accuracy": 0.8399689704179764, + "num_tokens": 191821.0, + "step": 20 + }, + { + "entropy": 0.28777916859835384, + "epoch": 0.0990916597853014, + "grad_norm": 12.375, + "learning_rate": 4.833333333333333e-06, + "loss": 0.7679028034210205, + "mean_token_accuracy": 0.8431851923465729, + "num_tokens": 285491.0, + "step": 30 + }, + { + "entropy": 0.3033197607845068, + "epoch": 0.13212221304706853, + "grad_norm": 5.375, + "learning_rate": 6.5000000000000004e-06, + "loss": 0.6137509822845459, + "mean_token_accuracy": 0.8549904882907867, + "num_tokens": 381273.0, + "step": 40 + }, + { + "entropy": 0.34702248610556125, + "epoch": 0.16515276630883569, + "grad_norm": 4.25, + "learning_rate": 8.166666666666668e-06, + "loss": 0.5155063152313233, + "mean_token_accuracy": 0.8607264116406441, + "num_tokens": 476481.0, + "step": 50 + }, + { + "entropy": 0.35821260381489994, + "epoch": 0.1981833195706028, + "grad_norm": 2.890625, + "learning_rate": 9.833333333333333e-06, + "loss": 0.42441272735595703, + "mean_token_accuracy": 0.8774531938135623, + "num_tokens": 570979.0, + "step": 60 + }, + { + "entropy": 0.33337059412151576, + "epoch": 0.23121387283236994, + "grad_norm": 3.15625, + "learning_rate": 9.835164835164835e-06, + "loss": 0.34101109504699706, + "mean_token_accuracy": 0.8939957685768605, + "num_tokens": 664789.0, + "step": 70 + }, + { + "entropy": 0.30876336991786957, + "epoch": 0.26424442609413706, + "grad_norm": 1.8515625, + "learning_rate": 9.652014652014653e-06, + "loss": 0.32317628860473635, + "mean_token_accuracy": 0.8987478196620942, + "num_tokens": 764246.0, + "step": 80 + }, + { + "entropy": 0.30824985895305873, + "epoch": 0.2972749793559042, + "grad_norm": 2.15625, + "learning_rate": 9.46886446886447e-06, + "loss": 0.31807353496551516, + "mean_token_accuracy": 0.896314675360918, + "num_tokens": 855689.0, + "step": 90 + }, + { + "entropy": 0.31522620394825934, + "epoch": 0.33030553261767137, + "grad_norm": 1.8828125, + "learning_rate": 9.285714285714288e-06, + "loss": 0.338785982131958, + "mean_token_accuracy": 0.89400485008955, + "num_tokens": 949366.0, + "step": 100 + }, + { + "epoch": 0.33030553261767137, + "eval_entropy": 0.3022541454544774, + "eval_loss": 0.3146008253097534, + "eval_mean_token_accuracy": 0.8988911988558592, + "eval_num_tokens": 949366.0, + "eval_runtime": 17.914, + "eval_samples_per_second": 30.088, + "eval_steps_per_second": 15.072, + "step": 100 + }, + { + "entropy": 0.2985268572345376, + "epoch": 0.3633360858794385, + "grad_norm": 2.171875, + "learning_rate": 9.102564102564104e-06, + "loss": 0.31417596340179443, + "mean_token_accuracy": 0.89812126532197, + "num_tokens": 1040854.0, + "step": 110 + }, + { + "entropy": 0.3040375892072916, + "epoch": 0.3963666391412056, + "grad_norm": 1.9609375, + "learning_rate": 8.91941391941392e-06, + "loss": 0.32050628662109376, + "mean_token_accuracy": 0.8970061101019382, + "num_tokens": 1132424.0, + "step": 120 + }, + { + "entropy": 0.3125588171184063, + "epoch": 0.42939719240297275, + "grad_norm": 1.8203125, + "learning_rate": 8.736263736263737e-06, + "loss": 0.3176266193389893, + "mean_token_accuracy": 0.8971590586006641, + "num_tokens": 1221899.0, + "step": 130 + }, + { + "entropy": 0.312872576713562, + "epoch": 0.4624277456647399, + "grad_norm": 1.78125, + "learning_rate": 8.553113553113553e-06, + "loss": 0.3343791484832764, + "mean_token_accuracy": 0.8940604299306869, + "num_tokens": 1315052.0, + "step": 140 + }, + { + "entropy": 0.27830475065857174, + "epoch": 0.495458298926507, + "grad_norm": 2.0, + "learning_rate": 8.369963369963371e-06, + "loss": 0.28718521595001223, + "mean_token_accuracy": 0.9051955968141556, + "num_tokens": 1401947.0, + "step": 150 + }, + { + "entropy": 0.3005422970280051, + "epoch": 0.5284888521882741, + "grad_norm": 2.34375, + "learning_rate": 8.186813186813188e-06, + "loss": 0.31569790840148926, + "mean_token_accuracy": 0.8999212823808194, + "num_tokens": 1494107.0, + "step": 160 + }, + { + "entropy": 0.31200682315975425, + "epoch": 0.5615194054500413, + "grad_norm": 2.015625, + "learning_rate": 8.003663003663006e-06, + "loss": 0.32325453758239747, + "mean_token_accuracy": 0.8952137432992459, + "num_tokens": 1586474.0, + "step": 170 + }, + { + "entropy": 0.2929861258715391, + "epoch": 0.5945499587118084, + "grad_norm": 2.09375, + "learning_rate": 7.820512820512822e-06, + "loss": 0.29693677425384524, + "mean_token_accuracy": 0.9031261466443539, + "num_tokens": 1678162.0, + "step": 180 + }, + { + "entropy": 0.30670158881694076, + "epoch": 0.6275805119735756, + "grad_norm": 1.828125, + "learning_rate": 7.637362637362638e-06, + "loss": 0.31567308902740476, + "mean_token_accuracy": 0.8983482263982296, + "num_tokens": 1781401.0, + "step": 190 + }, + { + "entropy": 0.2866023685783148, + "epoch": 0.6606110652353427, + "grad_norm": 1.6953125, + "learning_rate": 7.454212454212455e-06, + "loss": 0.29333360195159913, + "mean_token_accuracy": 0.904055442661047, + "num_tokens": 1868038.0, + "step": 200 + }, + { + "epoch": 0.6606110652353427, + "eval_entropy": 0.2921138784951634, + "eval_loss": 0.29292193055152893, + "eval_mean_token_accuracy": 0.904505420172656, + "eval_num_tokens": 1868038.0, + "eval_runtime": 17.8732, + "eval_samples_per_second": 30.157, + "eval_steps_per_second": 15.106, + "step": 200 + }, + { + "entropy": 0.28792863376438615, + "epoch": 0.6936416184971098, + "grad_norm": 1.9609375, + "learning_rate": 7.271062271062271e-06, + "loss": 0.29235827922821045, + "mean_token_accuracy": 0.9050419509410859, + "num_tokens": 1959819.0, + "step": 210 + }, + { + "entropy": 0.28098181951791046, + "epoch": 0.726672171758877, + "grad_norm": 1.8515625, + "learning_rate": 7.087912087912089e-06, + "loss": 0.2915766954421997, + "mean_token_accuracy": 0.9068245775997639, + "num_tokens": 2059113.0, + "step": 220 + }, + { + "entropy": 0.26028320472687483, + "epoch": 0.7597027250206441, + "grad_norm": 2.0, + "learning_rate": 6.9047619047619055e-06, + "loss": 0.2654874324798584, + "mean_token_accuracy": 0.9116183832287789, + "num_tokens": 2145321.0, + "step": 230 + }, + { + "entropy": 0.2855058180168271, + "epoch": 0.7927332782824112, + "grad_norm": 1.84375, + "learning_rate": 6.721611721611723e-06, + "loss": 0.2998212337493896, + "mean_token_accuracy": 0.9046589516103267, + "num_tokens": 2240859.0, + "step": 240 + }, + { + "entropy": 0.2783821925520897, + "epoch": 0.8257638315441783, + "grad_norm": 1.9609375, + "learning_rate": 6.538461538461539e-06, + "loss": 0.28870720863342286, + "mean_token_accuracy": 0.9073394626379013, + "num_tokens": 2333798.0, + "step": 250 + }, + { + "entropy": 0.277759512513876, + "epoch": 0.8587943848059455, + "grad_norm": 1.7734375, + "learning_rate": 6.3553113553113555e-06, + "loss": 0.29060728549957277, + "mean_token_accuracy": 0.9055352889001369, + "num_tokens": 2429910.0, + "step": 260 + }, + { + "entropy": 0.2882726760581136, + "epoch": 0.8918249380677127, + "grad_norm": 1.703125, + "learning_rate": 6.172161172161173e-06, + "loss": 0.2988502740859985, + "mean_token_accuracy": 0.9038613699376583, + "num_tokens": 2524730.0, + "step": 270 + }, + { + "entropy": 0.2855747708119452, + "epoch": 0.9248554913294798, + "grad_norm": 1.90625, + "learning_rate": 5.989010989010989e-06, + "loss": 0.289182186126709, + "mean_token_accuracy": 0.9049803033471108, + "num_tokens": 2615802.0, + "step": 280 + }, + { + "entropy": 0.26923007648438213, + "epoch": 0.9578860445912469, + "grad_norm": 2.84375, + "learning_rate": 5.805860805860807e-06, + "loss": 0.27834036350250246, + "mean_token_accuracy": 0.9109352678060532, + "num_tokens": 2709073.0, + "step": 290 + }, + { + "entropy": 0.2844470454379916, + "epoch": 0.990916597853014, + "grad_norm": 1.640625, + "learning_rate": 5.6227106227106235e-06, + "loss": 0.28873960971832274, + "mean_token_accuracy": 0.9069667845964432, + "num_tokens": 2805135.0, + "step": 300 + }, + { + "epoch": 0.990916597853014, + "eval_entropy": 0.285272175570329, + "eval_loss": 0.28705036640167236, + "eval_mean_token_accuracy": 0.9062359571456909, + "eval_num_tokens": 2805135.0, + "eval_runtime": 17.9038, + "eval_samples_per_second": 30.105, + "eval_steps_per_second": 15.081, + "step": 300 + }, + { + "entropy": 0.2869852078266633, + "epoch": 1.023121387283237, + "grad_norm": 1.9375, + "learning_rate": 5.43956043956044e-06, + "loss": 0.2850792646408081, + "mean_token_accuracy": 0.9064911672702203, + "num_tokens": 2898984.0, + "step": 310 + }, + { + "entropy": 0.2757460119202733, + "epoch": 1.0561519405450042, + "grad_norm": 1.8359375, + "learning_rate": 5.256410256410257e-06, + "loss": 0.2779810905456543, + "mean_token_accuracy": 0.9096895463764667, + "num_tokens": 2991694.0, + "step": 320 + }, + { + "entropy": 0.2600112119689584, + "epoch": 1.0891824938067713, + "grad_norm": 2.0625, + "learning_rate": 5.0732600732600735e-06, + "loss": 0.2662019729614258, + "mean_token_accuracy": 0.912333931028843, + "num_tokens": 3078862.0, + "step": 330 + }, + { + "entropy": 0.27342566605657337, + "epoch": 1.1222130470685383, + "grad_norm": 1.921875, + "learning_rate": 4.890109890109891e-06, + "loss": 0.28096842765808105, + "mean_token_accuracy": 0.9091624312102795, + "num_tokens": 3174256.0, + "step": 340 + }, + { + "entropy": 0.2671937083825469, + "epoch": 1.1552436003303055, + "grad_norm": 1.75, + "learning_rate": 4.706959706959707e-06, + "loss": 0.2723649501800537, + "mean_token_accuracy": 0.9101433955132961, + "num_tokens": 3272477.0, + "step": 350 + }, + { + "entropy": 0.2617026660591364, + "epoch": 1.1882741535920727, + "grad_norm": 1.9453125, + "learning_rate": 4.523809523809524e-06, + "loss": 0.26422107219696045, + "mean_token_accuracy": 0.913913157582283, + "num_tokens": 3364137.0, + "step": 360 + }, + { + "entropy": 0.2595691759139299, + "epoch": 1.2213047068538398, + "grad_norm": 1.5390625, + "learning_rate": 4.340659340659341e-06, + "loss": 0.2714951276779175, + "mean_token_accuracy": 0.911636833101511, + "num_tokens": 3460217.0, + "step": 370 + }, + { + "entropy": 0.2933320404961705, + "epoch": 1.254335260115607, + "grad_norm": 1.9765625, + "learning_rate": 4.157509157509158e-06, + "loss": 0.29573276042938235, + "mean_token_accuracy": 0.902192335575819, + "num_tokens": 3557030.0, + "step": 380 + }, + { + "entropy": 0.2693369995802641, + "epoch": 1.287365813377374, + "grad_norm": 1.890625, + "learning_rate": 3.974358974358974e-06, + "loss": 0.27785425186157225, + "mean_token_accuracy": 0.9090337291359901, + "num_tokens": 3654159.0, + "step": 390 + }, + { + "entropy": 0.2908114368095994, + "epoch": 1.3203963666391412, + "grad_norm": 2.015625, + "learning_rate": 3.7912087912087915e-06, + "loss": 0.30535905361175536, + "mean_token_accuracy": 0.901054035872221, + "num_tokens": 3749664.0, + "step": 400 + }, + { + "epoch": 1.3203963666391412, + "eval_entropy": 0.2783401116176888, + "eval_loss": 0.28495287895202637, + "eval_mean_token_accuracy": 0.9069356251645971, + "eval_num_tokens": 3749664.0, + "eval_runtime": 17.8814, + "eval_samples_per_second": 30.143, + "eval_steps_per_second": 15.099, + "step": 400 + }, + { + "entropy": 0.2749379588291049, + "epoch": 1.3534269199009084, + "grad_norm": 1.84375, + "learning_rate": 3.6080586080586083e-06, + "loss": 0.27795455455780027, + "mean_token_accuracy": 0.9090839587152004, + "num_tokens": 3845841.0, + "step": 410 + }, + { + "entropy": 0.2859474029392004, + "epoch": 1.3864574731626755, + "grad_norm": 1.6796875, + "learning_rate": 3.424908424908425e-06, + "loss": 0.2886682987213135, + "mean_token_accuracy": 0.9059950634837151, + "num_tokens": 3942402.0, + "step": 420 + }, + { + "entropy": 0.2537162031978369, + "epoch": 1.4194880264244425, + "grad_norm": 1.9375, + "learning_rate": 3.2417582417582424e-06, + "loss": 0.25533475875854494, + "mean_token_accuracy": 0.9158934533596039, + "num_tokens": 4034122.0, + "step": 430 + }, + { + "entropy": 0.28255065642297267, + "epoch": 1.4525185796862097, + "grad_norm": 1.7734375, + "learning_rate": 3.058608058608059e-06, + "loss": 0.29817631244659426, + "mean_token_accuracy": 0.9040707617998123, + "num_tokens": 4127068.0, + "step": 440 + }, + { + "entropy": 0.2898306529968977, + "epoch": 1.4855491329479769, + "grad_norm": 1.703125, + "learning_rate": 2.8754578754578755e-06, + "loss": 0.2923931121826172, + "mean_token_accuracy": 0.9030195340514183, + "num_tokens": 4224435.0, + "step": 450 + }, + { + "entropy": 0.27209411058574917, + "epoch": 1.518579686209744, + "grad_norm": 1.7578125, + "learning_rate": 2.6923076923076923e-06, + "loss": 0.28646528720855713, + "mean_token_accuracy": 0.9095881894230843, + "num_tokens": 4321028.0, + "step": 460 + }, + { + "entropy": 0.280727200768888, + "epoch": 1.5516102394715112, + "grad_norm": 1.7421875, + "learning_rate": 2.509157509157509e-06, + "loss": 0.2891413688659668, + "mean_token_accuracy": 0.9067636162042618, + "num_tokens": 4414916.0, + "step": 470 + }, + { + "entropy": 0.27109829243272543, + "epoch": 1.5846407927332784, + "grad_norm": 1.8359375, + "learning_rate": 2.3260073260073264e-06, + "loss": 0.26625516414642336, + "mean_token_accuracy": 0.9099020563066006, + "num_tokens": 4505708.0, + "step": 480 + }, + { + "entropy": 0.2729395367205143, + "epoch": 1.6176713459950454, + "grad_norm": 1.7890625, + "learning_rate": 2.1428571428571427e-06, + "loss": 0.2779061317443848, + "mean_token_accuracy": 0.909747340530157, + "num_tokens": 4600244.0, + "step": 490 + }, + { + "entropy": 0.2682301852852106, + "epoch": 1.6507018992568125, + "grad_norm": 1.6953125, + "learning_rate": 1.95970695970696e-06, + "loss": 0.274446964263916, + "mean_token_accuracy": 0.9094639919698239, + "num_tokens": 4688147.0, + "step": 500 + }, + { + "epoch": 1.6507018992568125, + "eval_entropy": 0.2756172451708052, + "eval_loss": 0.2841346263885498, + "eval_mean_token_accuracy": 0.9072470570052111, + "eval_num_tokens": 4688147.0, + "eval_runtime": 17.9119, + "eval_samples_per_second": 30.092, + "eval_steps_per_second": 15.074, + "step": 500 + } + ], + "logging_steps": 10, + "max_steps": 606, + "num_input_tokens_seen": 0, + "num_train_epochs": 2, + "save_steps": 100, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": false + }, + "attributes": {} + } + }, + "total_flos": 5.028742352950886e+16, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-500/training_args.bin b/checkpoint-500/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..133cb6661b0124bb9db0d7d208d748fd736ea689 --- /dev/null +++ b/checkpoint-500/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6696e1fef8bde5e22de2f5ae18c284f2b2d35a345157db5151571518385fb38b +size 5713 diff --git a/checkpoint-600/chat_template.jinja b/checkpoint-600/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-600/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-600/config.json b/checkpoint-600/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-600/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-600/generation_config.json b/checkpoint-600/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-600/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-600/model.safetensors b/checkpoint-600/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..888c633f416471cd0dba703e4ed15cd967deaf00 --- /dev/null +++ b/checkpoint-600/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:066f20885c5c9a598d45fbcd21aa7ab9081dffda3fb1930d573d7c3d99b5d998 +size 3441185608 diff --git a/checkpoint-600/optimizer.pt b/checkpoint-600/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..315889bc3c6c060ec3677e282757036eb6d14155 --- /dev/null +++ b/checkpoint-600/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c75c57fe3c14c29f7a73acb2b44e60573aaf69cd744a4ce99770cdfdbe4de45e +size 6882572207 diff --git a/checkpoint-600/rng_state.pth b/checkpoint-600/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..1fb544a66454d577cf2a3cb06c26493972f57952 --- /dev/null +++ b/checkpoint-600/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fdc37bbd2e979f041dfbbb004a5c74bab6cdda159cb18116df728588515a9ef6 +size 14645 diff --git a/checkpoint-600/scheduler.pt b/checkpoint-600/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..e2645e63119eba5c7c9361da38b055b60e6d8985 --- /dev/null +++ b/checkpoint-600/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:540d8ab45fe72a01988766f9263355239d34a4ff0ef13ddac7f1f0d1198ce9a9 +size 1465 diff --git a/checkpoint-600/tokenizer.json b/checkpoint-600/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-600/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-600/tokenizer_config.json b/checkpoint-600/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-600/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-600/trainer_state.json b/checkpoint-600/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..c4cf7cbe351f817a25fffdf26360f8e89d0ea1ce --- /dev/null +++ b/checkpoint-600/trainer_state.json @@ -0,0 +1,700 @@ +{ + "best_global_step": 600, + "best_metric": 0.28394654393196106, + "best_model_checkpoint": "/scratch/checkpoints/math_sft_v2/checkpoint-600", + "epoch": 1.981007431874484, + "eval_steps": 100, + "global_step": 600, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": 0.26208699569106103, + "epoch": 0.03303055326176713, + "grad_norm": 13.3125, + "learning_rate": 1.5e-06, + "loss": 0.8376299858093261, + "mean_token_accuracy": 0.8421687923371792, + "num_tokens": 98215.0, + "step": 10 + }, + { + "entropy": 0.2689803125336766, + "epoch": 0.06606110652353427, + "grad_norm": 11.5625, + "learning_rate": 3.1666666666666667e-06, + "loss": 0.853145980834961, + "mean_token_accuracy": 0.8399689704179764, + "num_tokens": 191821.0, + "step": 20 + }, + { + "entropy": 0.28777916859835384, + "epoch": 0.0990916597853014, + "grad_norm": 12.375, + "learning_rate": 4.833333333333333e-06, + "loss": 0.7679028034210205, + "mean_token_accuracy": 0.8431851923465729, + "num_tokens": 285491.0, + "step": 30 + }, + { + "entropy": 0.3033197607845068, + "epoch": 0.13212221304706853, + "grad_norm": 5.375, + "learning_rate": 6.5000000000000004e-06, + "loss": 0.6137509822845459, + "mean_token_accuracy": 0.8549904882907867, + "num_tokens": 381273.0, + "step": 40 + }, + { + "entropy": 0.34702248610556125, + "epoch": 0.16515276630883569, + "grad_norm": 4.25, + "learning_rate": 8.166666666666668e-06, + "loss": 0.5155063152313233, + "mean_token_accuracy": 0.8607264116406441, + "num_tokens": 476481.0, + "step": 50 + }, + { + "entropy": 0.35821260381489994, + "epoch": 0.1981833195706028, + "grad_norm": 2.890625, + "learning_rate": 9.833333333333333e-06, + "loss": 0.42441272735595703, + "mean_token_accuracy": 0.8774531938135623, + "num_tokens": 570979.0, + "step": 60 + }, + { + "entropy": 0.33337059412151576, + "epoch": 0.23121387283236994, + "grad_norm": 3.15625, + "learning_rate": 9.835164835164835e-06, + "loss": 0.34101109504699706, + "mean_token_accuracy": 0.8939957685768605, + "num_tokens": 664789.0, + "step": 70 + }, + { + "entropy": 0.30876336991786957, + "epoch": 0.26424442609413706, + "grad_norm": 1.8515625, + "learning_rate": 9.652014652014653e-06, + "loss": 0.32317628860473635, + "mean_token_accuracy": 0.8987478196620942, + "num_tokens": 764246.0, + "step": 80 + }, + { + "entropy": 0.30824985895305873, + "epoch": 0.2972749793559042, + "grad_norm": 2.15625, + "learning_rate": 9.46886446886447e-06, + "loss": 0.31807353496551516, + "mean_token_accuracy": 0.896314675360918, + "num_tokens": 855689.0, + "step": 90 + }, + { + "entropy": 0.31522620394825934, + "epoch": 0.33030553261767137, + "grad_norm": 1.8828125, + "learning_rate": 9.285714285714288e-06, + "loss": 0.338785982131958, + "mean_token_accuracy": 0.89400485008955, + "num_tokens": 949366.0, + "step": 100 + }, + { + "epoch": 0.33030553261767137, + "eval_entropy": 0.3022541454544774, + "eval_loss": 0.3146008253097534, + "eval_mean_token_accuracy": 0.8988911988558592, + "eval_num_tokens": 949366.0, + "eval_runtime": 17.914, + "eval_samples_per_second": 30.088, + "eval_steps_per_second": 15.072, + "step": 100 + }, + { + "entropy": 0.2985268572345376, + "epoch": 0.3633360858794385, + "grad_norm": 2.171875, + "learning_rate": 9.102564102564104e-06, + "loss": 0.31417596340179443, + "mean_token_accuracy": 0.89812126532197, + "num_tokens": 1040854.0, + "step": 110 + }, + { + "entropy": 0.3040375892072916, + "epoch": 0.3963666391412056, + "grad_norm": 1.9609375, + "learning_rate": 8.91941391941392e-06, + "loss": 0.32050628662109376, + "mean_token_accuracy": 0.8970061101019382, + "num_tokens": 1132424.0, + "step": 120 + }, + { + "entropy": 0.3125588171184063, + "epoch": 0.42939719240297275, + "grad_norm": 1.8203125, + "learning_rate": 8.736263736263737e-06, + "loss": 0.3176266193389893, + "mean_token_accuracy": 0.8971590586006641, + "num_tokens": 1221899.0, + "step": 130 + }, + { + "entropy": 0.312872576713562, + "epoch": 0.4624277456647399, + "grad_norm": 1.78125, + "learning_rate": 8.553113553113553e-06, + "loss": 0.3343791484832764, + "mean_token_accuracy": 0.8940604299306869, + "num_tokens": 1315052.0, + "step": 140 + }, + { + "entropy": 0.27830475065857174, + "epoch": 0.495458298926507, + "grad_norm": 2.0, + "learning_rate": 8.369963369963371e-06, + "loss": 0.28718521595001223, + "mean_token_accuracy": 0.9051955968141556, + "num_tokens": 1401947.0, + "step": 150 + }, + { + "entropy": 0.3005422970280051, + "epoch": 0.5284888521882741, + "grad_norm": 2.34375, + "learning_rate": 8.186813186813188e-06, + "loss": 0.31569790840148926, + "mean_token_accuracy": 0.8999212823808194, + "num_tokens": 1494107.0, + "step": 160 + }, + { + "entropy": 0.31200682315975425, + "epoch": 0.5615194054500413, + "grad_norm": 2.015625, + "learning_rate": 8.003663003663006e-06, + "loss": 0.32325453758239747, + "mean_token_accuracy": 0.8952137432992459, + "num_tokens": 1586474.0, + "step": 170 + }, + { + "entropy": 0.2929861258715391, + "epoch": 0.5945499587118084, + "grad_norm": 2.09375, + "learning_rate": 7.820512820512822e-06, + "loss": 0.29693677425384524, + "mean_token_accuracy": 0.9031261466443539, + "num_tokens": 1678162.0, + "step": 180 + }, + { + "entropy": 0.30670158881694076, + "epoch": 0.6275805119735756, + "grad_norm": 1.828125, + "learning_rate": 7.637362637362638e-06, + "loss": 0.31567308902740476, + "mean_token_accuracy": 0.8983482263982296, + "num_tokens": 1781401.0, + "step": 190 + }, + { + "entropy": 0.2866023685783148, + "epoch": 0.6606110652353427, + "grad_norm": 1.6953125, + "learning_rate": 7.454212454212455e-06, + "loss": 0.29333360195159913, + "mean_token_accuracy": 0.904055442661047, + "num_tokens": 1868038.0, + "step": 200 + }, + { + "epoch": 0.6606110652353427, + "eval_entropy": 0.2921138784951634, + "eval_loss": 0.29292193055152893, + "eval_mean_token_accuracy": 0.904505420172656, + "eval_num_tokens": 1868038.0, + "eval_runtime": 17.8732, + "eval_samples_per_second": 30.157, + "eval_steps_per_second": 15.106, + "step": 200 + }, + { + "entropy": 0.28792863376438615, + "epoch": 0.6936416184971098, + "grad_norm": 1.9609375, + "learning_rate": 7.271062271062271e-06, + "loss": 0.29235827922821045, + "mean_token_accuracy": 0.9050419509410859, + "num_tokens": 1959819.0, + "step": 210 + }, + { + "entropy": 0.28098181951791046, + "epoch": 0.726672171758877, + "grad_norm": 1.8515625, + "learning_rate": 7.087912087912089e-06, + "loss": 0.2915766954421997, + "mean_token_accuracy": 0.9068245775997639, + "num_tokens": 2059113.0, + "step": 220 + }, + { + "entropy": 0.26028320472687483, + "epoch": 0.7597027250206441, + "grad_norm": 2.0, + "learning_rate": 6.9047619047619055e-06, + "loss": 0.2654874324798584, + "mean_token_accuracy": 0.9116183832287789, + "num_tokens": 2145321.0, + "step": 230 + }, + { + "entropy": 0.2855058180168271, + "epoch": 0.7927332782824112, + "grad_norm": 1.84375, + "learning_rate": 6.721611721611723e-06, + "loss": 0.2998212337493896, + "mean_token_accuracy": 0.9046589516103267, + "num_tokens": 2240859.0, + "step": 240 + }, + { + "entropy": 0.2783821925520897, + "epoch": 0.8257638315441783, + "grad_norm": 1.9609375, + "learning_rate": 6.538461538461539e-06, + "loss": 0.28870720863342286, + "mean_token_accuracy": 0.9073394626379013, + "num_tokens": 2333798.0, + "step": 250 + }, + { + "entropy": 0.277759512513876, + "epoch": 0.8587943848059455, + "grad_norm": 1.7734375, + "learning_rate": 6.3553113553113555e-06, + "loss": 0.29060728549957277, + "mean_token_accuracy": 0.9055352889001369, + "num_tokens": 2429910.0, + "step": 260 + }, + { + "entropy": 0.2882726760581136, + "epoch": 0.8918249380677127, + "grad_norm": 1.703125, + "learning_rate": 6.172161172161173e-06, + "loss": 0.2988502740859985, + "mean_token_accuracy": 0.9038613699376583, + "num_tokens": 2524730.0, + "step": 270 + }, + { + "entropy": 0.2855747708119452, + "epoch": 0.9248554913294798, + "grad_norm": 1.90625, + "learning_rate": 5.989010989010989e-06, + "loss": 0.289182186126709, + "mean_token_accuracy": 0.9049803033471108, + "num_tokens": 2615802.0, + "step": 280 + }, + { + "entropy": 0.26923007648438213, + "epoch": 0.9578860445912469, + "grad_norm": 2.84375, + "learning_rate": 5.805860805860807e-06, + "loss": 0.27834036350250246, + "mean_token_accuracy": 0.9109352678060532, + "num_tokens": 2709073.0, + "step": 290 + }, + { + "entropy": 0.2844470454379916, + "epoch": 0.990916597853014, + "grad_norm": 1.640625, + "learning_rate": 5.6227106227106235e-06, + "loss": 0.28873960971832274, + "mean_token_accuracy": 0.9069667845964432, + "num_tokens": 2805135.0, + "step": 300 + }, + { + "epoch": 0.990916597853014, + "eval_entropy": 0.285272175570329, + "eval_loss": 0.28705036640167236, + "eval_mean_token_accuracy": 0.9062359571456909, + "eval_num_tokens": 2805135.0, + "eval_runtime": 17.9038, + "eval_samples_per_second": 30.105, + "eval_steps_per_second": 15.081, + "step": 300 + }, + { + "entropy": 0.2869852078266633, + "epoch": 1.023121387283237, + "grad_norm": 1.9375, + "learning_rate": 5.43956043956044e-06, + "loss": 0.2850792646408081, + "mean_token_accuracy": 0.9064911672702203, + "num_tokens": 2898984.0, + "step": 310 + }, + { + "entropy": 0.2757460119202733, + "epoch": 1.0561519405450042, + "grad_norm": 1.8359375, + "learning_rate": 5.256410256410257e-06, + "loss": 0.2779810905456543, + "mean_token_accuracy": 0.9096895463764667, + "num_tokens": 2991694.0, + "step": 320 + }, + { + "entropy": 0.2600112119689584, + "epoch": 1.0891824938067713, + "grad_norm": 2.0625, + "learning_rate": 5.0732600732600735e-06, + "loss": 0.2662019729614258, + "mean_token_accuracy": 0.912333931028843, + "num_tokens": 3078862.0, + "step": 330 + }, + { + "entropy": 0.27342566605657337, + "epoch": 1.1222130470685383, + "grad_norm": 1.921875, + "learning_rate": 4.890109890109891e-06, + "loss": 0.28096842765808105, + "mean_token_accuracy": 0.9091624312102795, + "num_tokens": 3174256.0, + "step": 340 + }, + { + "entropy": 0.2671937083825469, + "epoch": 1.1552436003303055, + "grad_norm": 1.75, + "learning_rate": 4.706959706959707e-06, + "loss": 0.2723649501800537, + "mean_token_accuracy": 0.9101433955132961, + "num_tokens": 3272477.0, + "step": 350 + }, + { + "entropy": 0.2617026660591364, + "epoch": 1.1882741535920727, + "grad_norm": 1.9453125, + "learning_rate": 4.523809523809524e-06, + "loss": 0.26422107219696045, + "mean_token_accuracy": 0.913913157582283, + "num_tokens": 3364137.0, + "step": 360 + }, + { + "entropy": 0.2595691759139299, + "epoch": 1.2213047068538398, + "grad_norm": 1.5390625, + "learning_rate": 4.340659340659341e-06, + "loss": 0.2714951276779175, + "mean_token_accuracy": 0.911636833101511, + "num_tokens": 3460217.0, + "step": 370 + }, + { + "entropy": 0.2933320404961705, + "epoch": 1.254335260115607, + "grad_norm": 1.9765625, + "learning_rate": 4.157509157509158e-06, + "loss": 0.29573276042938235, + "mean_token_accuracy": 0.902192335575819, + "num_tokens": 3557030.0, + "step": 380 + }, + { + "entropy": 0.2693369995802641, + "epoch": 1.287365813377374, + "grad_norm": 1.890625, + "learning_rate": 3.974358974358974e-06, + "loss": 0.27785425186157225, + "mean_token_accuracy": 0.9090337291359901, + "num_tokens": 3654159.0, + "step": 390 + }, + { + "entropy": 0.2908114368095994, + "epoch": 1.3203963666391412, + "grad_norm": 2.015625, + "learning_rate": 3.7912087912087915e-06, + "loss": 0.30535905361175536, + "mean_token_accuracy": 0.901054035872221, + "num_tokens": 3749664.0, + "step": 400 + }, + { + "epoch": 1.3203963666391412, + "eval_entropy": 0.2783401116176888, + "eval_loss": 0.28495287895202637, + "eval_mean_token_accuracy": 0.9069356251645971, + "eval_num_tokens": 3749664.0, + "eval_runtime": 17.8814, + "eval_samples_per_second": 30.143, + "eval_steps_per_second": 15.099, + "step": 400 + }, + { + "entropy": 0.2749379588291049, + "epoch": 1.3534269199009084, + "grad_norm": 1.84375, + "learning_rate": 3.6080586080586083e-06, + "loss": 0.27795455455780027, + "mean_token_accuracy": 0.9090839587152004, + "num_tokens": 3845841.0, + "step": 410 + }, + { + "entropy": 0.2859474029392004, + "epoch": 1.3864574731626755, + "grad_norm": 1.6796875, + "learning_rate": 3.424908424908425e-06, + "loss": 0.2886682987213135, + "mean_token_accuracy": 0.9059950634837151, + "num_tokens": 3942402.0, + "step": 420 + }, + { + "entropy": 0.2537162031978369, + "epoch": 1.4194880264244425, + "grad_norm": 1.9375, + "learning_rate": 3.2417582417582424e-06, + "loss": 0.25533475875854494, + "mean_token_accuracy": 0.9158934533596039, + "num_tokens": 4034122.0, + "step": 430 + }, + { + "entropy": 0.28255065642297267, + "epoch": 1.4525185796862097, + "grad_norm": 1.7734375, + "learning_rate": 3.058608058608059e-06, + "loss": 0.29817631244659426, + "mean_token_accuracy": 0.9040707617998123, + "num_tokens": 4127068.0, + "step": 440 + }, + { + "entropy": 0.2898306529968977, + "epoch": 1.4855491329479769, + "grad_norm": 1.703125, + "learning_rate": 2.8754578754578755e-06, + "loss": 0.2923931121826172, + "mean_token_accuracy": 0.9030195340514183, + "num_tokens": 4224435.0, + "step": 450 + }, + { + "entropy": 0.27209411058574917, + "epoch": 1.518579686209744, + "grad_norm": 1.7578125, + "learning_rate": 2.6923076923076923e-06, + "loss": 0.28646528720855713, + "mean_token_accuracy": 0.9095881894230843, + "num_tokens": 4321028.0, + "step": 460 + }, + { + "entropy": 0.280727200768888, + "epoch": 1.5516102394715112, + "grad_norm": 1.7421875, + "learning_rate": 2.509157509157509e-06, + "loss": 0.2891413688659668, + "mean_token_accuracy": 0.9067636162042618, + "num_tokens": 4414916.0, + "step": 470 + }, + { + "entropy": 0.27109829243272543, + "epoch": 1.5846407927332784, + "grad_norm": 1.8359375, + "learning_rate": 2.3260073260073264e-06, + "loss": 0.26625516414642336, + "mean_token_accuracy": 0.9099020563066006, + "num_tokens": 4505708.0, + "step": 480 + }, + { + "entropy": 0.2729395367205143, + "epoch": 1.6176713459950454, + "grad_norm": 1.7890625, + "learning_rate": 2.1428571428571427e-06, + "loss": 0.2779061317443848, + "mean_token_accuracy": 0.909747340530157, + "num_tokens": 4600244.0, + "step": 490 + }, + { + "entropy": 0.2682301852852106, + "epoch": 1.6507018992568125, + "grad_norm": 1.6953125, + "learning_rate": 1.95970695970696e-06, + "loss": 0.274446964263916, + "mean_token_accuracy": 0.9094639919698239, + "num_tokens": 4688147.0, + "step": 500 + }, + { + "epoch": 1.6507018992568125, + "eval_entropy": 0.2756172451708052, + "eval_loss": 0.2841346263885498, + "eval_mean_token_accuracy": 0.9072470570052111, + "eval_num_tokens": 4688147.0, + "eval_runtime": 17.9119, + "eval_samples_per_second": 30.092, + "eval_steps_per_second": 15.074, + "step": 500 + }, + { + "entropy": 0.26342453975230457, + "epoch": 1.6837324525185797, + "grad_norm": 1.890625, + "learning_rate": 1.7765567765567768e-06, + "loss": 0.26432271003723146, + "mean_token_accuracy": 0.9114409819245338, + "num_tokens": 4780124.0, + "step": 510 + }, + { + "entropy": 0.28657868495211003, + "epoch": 1.7167630057803467, + "grad_norm": 1.7734375, + "learning_rate": 1.5934065934065933e-06, + "loss": 0.2983593225479126, + "mean_token_accuracy": 0.9028328903019428, + "num_tokens": 4874633.0, + "step": 520 + }, + { + "entropy": 0.25633321776986123, + "epoch": 1.7497935590421139, + "grad_norm": 1.8359375, + "learning_rate": 1.4102564102564104e-06, + "loss": 0.2606488227844238, + "mean_token_accuracy": 0.9153999648988247, + "num_tokens": 4967263.0, + "step": 530 + }, + { + "entropy": 0.26120030507445335, + "epoch": 1.782824112303881, + "grad_norm": 2.078125, + "learning_rate": 1.2271062271062271e-06, + "loss": 0.2706313610076904, + "mean_token_accuracy": 0.9130682416260243, + "num_tokens": 5056736.0, + "step": 540 + }, + { + "entropy": 0.27009565513581035, + "epoch": 1.8158546655656482, + "grad_norm": 1.8828125, + "learning_rate": 1.0439560439560442e-06, + "loss": 0.277858567237854, + "mean_token_accuracy": 0.9087999157607556, + "num_tokens": 5150617.0, + "step": 550 + }, + { + "entropy": 0.2819518899545074, + "epoch": 1.8488852188274154, + "grad_norm": 1.9609375, + "learning_rate": 8.608058608058609e-07, + "loss": 0.28743035793304444, + "mean_token_accuracy": 0.9067244723439216, + "num_tokens": 5243538.0, + "step": 560 + }, + { + "entropy": 0.2858045116066933, + "epoch": 1.8819157720891826, + "grad_norm": 2.203125, + "learning_rate": 6.776556776556777e-07, + "loss": 0.2939825296401978, + "mean_token_accuracy": 0.904579871147871, + "num_tokens": 5331441.0, + "step": 570 + }, + { + "entropy": 0.2926233192905784, + "epoch": 1.9149463253509498, + "grad_norm": 2.3125, + "learning_rate": 4.945054945054946e-07, + "loss": 0.29961700439453126, + "mean_token_accuracy": 0.9033655568957328, + "num_tokens": 5426734.0, + "step": 580 + }, + { + "entropy": 0.2706678254529834, + "epoch": 1.9479768786127167, + "grad_norm": 2.03125, + "learning_rate": 3.113553113553114e-07, + "loss": 0.27953526973724363, + "mean_token_accuracy": 0.9093762949109078, + "num_tokens": 5518190.0, + "step": 590 + }, + { + "entropy": 0.27056904807686805, + "epoch": 1.981007431874484, + "grad_norm": 2.09375, + "learning_rate": 1.282051282051282e-07, + "loss": 0.2764536619186401, + "mean_token_accuracy": 0.9110228553414345, + "num_tokens": 5609238.0, + "step": 600 + }, + { + "epoch": 1.981007431874484, + "eval_entropy": 0.2757842331572815, + "eval_loss": 0.28394654393196106, + "eval_mean_token_accuracy": 0.9073928307603907, + "eval_num_tokens": 5609238.0, + "eval_runtime": 18.1138, + "eval_samples_per_second": 29.756, + "eval_steps_per_second": 14.906, + "step": 600 + } + ], + "logging_steps": 10, + "max_steps": 606, + "num_input_tokens_seen": 0, + "num_train_epochs": 2, + "save_steps": 100, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": false + }, + "attributes": {} + } + }, + "total_flos": 6.018332557481165e+16, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-600/training_args.bin b/checkpoint-600/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..133cb6661b0124bb9db0d7d208d748fd736ea689 --- /dev/null +++ b/checkpoint-600/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6696e1fef8bde5e22de2f5ae18c284f2b2d35a345157db5151571518385fb38b +size 5713 diff --git a/checkpoint-606/chat_template.jinja b/checkpoint-606/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-606/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-606/config.json b/checkpoint-606/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-606/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-606/generation_config.json b/checkpoint-606/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-606/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-606/model.safetensors b/checkpoint-606/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b08d1501dfdc93522108efc90dd1cd038d65cc7a --- /dev/null +++ b/checkpoint-606/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b2e0bbbd38e2572fcb7a7cdd2bf58165f2cb2a115544c68b132848cd7973d106 +size 3441185608 diff --git a/checkpoint-606/optimizer.pt b/checkpoint-606/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..590b9fb7b131560c1b58842b49379b273c5a6449 --- /dev/null +++ b/checkpoint-606/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:bc3d8578da8d88239c2024e2dd1da1012e2173f003e746efcc15b17915d0a03c +size 6882572207 diff --git a/checkpoint-606/rng_state.pth b/checkpoint-606/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..eed58b88838bddfd46b410984c9cbddf403b1437 --- /dev/null +++ b/checkpoint-606/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:363c5df1543d2c82b2f13164f35bdd0367ceb32e7fa1b2f67c19df073a08b17b +size 14645 diff --git a/checkpoint-606/scheduler.pt b/checkpoint-606/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..f182dc5e2546b5fdccbdc22b90eb16a565c67e6c --- /dev/null +++ b/checkpoint-606/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ef108e31778ee002976f58f4a1ef7b3f1228ac6f0b9152b3822e08d82d99376b +size 1465 diff --git a/checkpoint-606/tokenizer.json b/checkpoint-606/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-606/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-606/tokenizer_config.json b/checkpoint-606/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-606/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-606/trainer_state.json b/checkpoint-606/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..a4ec67787e726511e6fbf586ff5bd33e585236d7 --- /dev/null +++ b/checkpoint-606/trainer_state.json @@ -0,0 +1,711 @@ +{ + "best_global_step": 600, + "best_metric": 0.28394654393196106, + "best_model_checkpoint": "/scratch/checkpoints/math_sft_v2/checkpoint-600", + "epoch": 2.0, + "eval_steps": 100, + "global_step": 606, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": 0.26208699569106103, + "epoch": 0.03303055326176713, + "grad_norm": 13.3125, + "learning_rate": 1.5e-06, + "loss": 0.8376299858093261, + "mean_token_accuracy": 0.8421687923371792, + "num_tokens": 98215.0, + "step": 10 + }, + { + "entropy": 0.2689803125336766, + "epoch": 0.06606110652353427, + "grad_norm": 11.5625, + "learning_rate": 3.1666666666666667e-06, + "loss": 0.853145980834961, + "mean_token_accuracy": 0.8399689704179764, + "num_tokens": 191821.0, + "step": 20 + }, + { + "entropy": 0.28777916859835384, + "epoch": 0.0990916597853014, + "grad_norm": 12.375, + "learning_rate": 4.833333333333333e-06, + "loss": 0.7679028034210205, + "mean_token_accuracy": 0.8431851923465729, + "num_tokens": 285491.0, + "step": 30 + }, + { + "entropy": 0.3033197607845068, + "epoch": 0.13212221304706853, + "grad_norm": 5.375, + "learning_rate": 6.5000000000000004e-06, + "loss": 0.6137509822845459, + "mean_token_accuracy": 0.8549904882907867, + "num_tokens": 381273.0, + "step": 40 + }, + { + "entropy": 0.34702248610556125, + "epoch": 0.16515276630883569, + "grad_norm": 4.25, + "learning_rate": 8.166666666666668e-06, + "loss": 0.5155063152313233, + "mean_token_accuracy": 0.8607264116406441, + "num_tokens": 476481.0, + "step": 50 + }, + { + "entropy": 0.35821260381489994, + "epoch": 0.1981833195706028, + "grad_norm": 2.890625, + "learning_rate": 9.833333333333333e-06, + "loss": 0.42441272735595703, + "mean_token_accuracy": 0.8774531938135623, + "num_tokens": 570979.0, + "step": 60 + }, + { + "entropy": 0.33337059412151576, + "epoch": 0.23121387283236994, + "grad_norm": 3.15625, + "learning_rate": 9.835164835164835e-06, + "loss": 0.34101109504699706, + "mean_token_accuracy": 0.8939957685768605, + "num_tokens": 664789.0, + "step": 70 + }, + { + "entropy": 0.30876336991786957, + "epoch": 0.26424442609413706, + "grad_norm": 1.8515625, + "learning_rate": 9.652014652014653e-06, + "loss": 0.32317628860473635, + "mean_token_accuracy": 0.8987478196620942, + "num_tokens": 764246.0, + "step": 80 + }, + { + "entropy": 0.30824985895305873, + "epoch": 0.2972749793559042, + "grad_norm": 2.15625, + "learning_rate": 9.46886446886447e-06, + "loss": 0.31807353496551516, + "mean_token_accuracy": 0.896314675360918, + "num_tokens": 855689.0, + "step": 90 + }, + { + "entropy": 0.31522620394825934, + "epoch": 0.33030553261767137, + "grad_norm": 1.8828125, + "learning_rate": 9.285714285714288e-06, + "loss": 0.338785982131958, + "mean_token_accuracy": 0.89400485008955, + "num_tokens": 949366.0, + "step": 100 + }, + { + "epoch": 0.33030553261767137, + "eval_entropy": 0.3022541454544774, + "eval_loss": 0.3146008253097534, + "eval_mean_token_accuracy": 0.8988911988558592, + "eval_num_tokens": 949366.0, + "eval_runtime": 17.914, + "eval_samples_per_second": 30.088, + "eval_steps_per_second": 15.072, + "step": 100 + }, + { + "entropy": 0.2985268572345376, + "epoch": 0.3633360858794385, + "grad_norm": 2.171875, + "learning_rate": 9.102564102564104e-06, + "loss": 0.31417596340179443, + "mean_token_accuracy": 0.89812126532197, + "num_tokens": 1040854.0, + "step": 110 + }, + { + "entropy": 0.3040375892072916, + "epoch": 0.3963666391412056, + "grad_norm": 1.9609375, + "learning_rate": 8.91941391941392e-06, + "loss": 0.32050628662109376, + "mean_token_accuracy": 0.8970061101019382, + "num_tokens": 1132424.0, + "step": 120 + }, + { + "entropy": 0.3125588171184063, + "epoch": 0.42939719240297275, + "grad_norm": 1.8203125, + "learning_rate": 8.736263736263737e-06, + "loss": 0.3176266193389893, + "mean_token_accuracy": 0.8971590586006641, + "num_tokens": 1221899.0, + "step": 130 + }, + { + "entropy": 0.312872576713562, + "epoch": 0.4624277456647399, + "grad_norm": 1.78125, + "learning_rate": 8.553113553113553e-06, + "loss": 0.3343791484832764, + "mean_token_accuracy": 0.8940604299306869, + "num_tokens": 1315052.0, + "step": 140 + }, + { + "entropy": 0.27830475065857174, + "epoch": 0.495458298926507, + "grad_norm": 2.0, + "learning_rate": 8.369963369963371e-06, + "loss": 0.28718521595001223, + "mean_token_accuracy": 0.9051955968141556, + "num_tokens": 1401947.0, + "step": 150 + }, + { + "entropy": 0.3005422970280051, + "epoch": 0.5284888521882741, + "grad_norm": 2.34375, + "learning_rate": 8.186813186813188e-06, + "loss": 0.31569790840148926, + "mean_token_accuracy": 0.8999212823808194, + "num_tokens": 1494107.0, + "step": 160 + }, + { + "entropy": 0.31200682315975425, + "epoch": 0.5615194054500413, + "grad_norm": 2.015625, + "learning_rate": 8.003663003663006e-06, + "loss": 0.32325453758239747, + "mean_token_accuracy": 0.8952137432992459, + "num_tokens": 1586474.0, + "step": 170 + }, + { + "entropy": 0.2929861258715391, + "epoch": 0.5945499587118084, + "grad_norm": 2.09375, + "learning_rate": 7.820512820512822e-06, + "loss": 0.29693677425384524, + "mean_token_accuracy": 0.9031261466443539, + "num_tokens": 1678162.0, + "step": 180 + }, + { + "entropy": 0.30670158881694076, + "epoch": 0.6275805119735756, + "grad_norm": 1.828125, + "learning_rate": 7.637362637362638e-06, + "loss": 0.31567308902740476, + "mean_token_accuracy": 0.8983482263982296, + "num_tokens": 1781401.0, + "step": 190 + }, + { + "entropy": 0.2866023685783148, + "epoch": 0.6606110652353427, + "grad_norm": 1.6953125, + "learning_rate": 7.454212454212455e-06, + "loss": 0.29333360195159913, + "mean_token_accuracy": 0.904055442661047, + "num_tokens": 1868038.0, + "step": 200 + }, + { + "epoch": 0.6606110652353427, + "eval_entropy": 0.2921138784951634, + "eval_loss": 0.29292193055152893, + "eval_mean_token_accuracy": 0.904505420172656, + "eval_num_tokens": 1868038.0, + "eval_runtime": 17.8732, + "eval_samples_per_second": 30.157, + "eval_steps_per_second": 15.106, + "step": 200 + }, + { + "entropy": 0.28792863376438615, + "epoch": 0.6936416184971098, + "grad_norm": 1.9609375, + "learning_rate": 7.271062271062271e-06, + "loss": 0.29235827922821045, + "mean_token_accuracy": 0.9050419509410859, + "num_tokens": 1959819.0, + "step": 210 + }, + { + "entropy": 0.28098181951791046, + "epoch": 0.726672171758877, + "grad_norm": 1.8515625, + "learning_rate": 7.087912087912089e-06, + "loss": 0.2915766954421997, + "mean_token_accuracy": 0.9068245775997639, + "num_tokens": 2059113.0, + "step": 220 + }, + { + "entropy": 0.26028320472687483, + "epoch": 0.7597027250206441, + "grad_norm": 2.0, + "learning_rate": 6.9047619047619055e-06, + "loss": 0.2654874324798584, + "mean_token_accuracy": 0.9116183832287789, + "num_tokens": 2145321.0, + "step": 230 + }, + { + "entropy": 0.2855058180168271, + "epoch": 0.7927332782824112, + "grad_norm": 1.84375, + "learning_rate": 6.721611721611723e-06, + "loss": 0.2998212337493896, + "mean_token_accuracy": 0.9046589516103267, + "num_tokens": 2240859.0, + "step": 240 + }, + { + "entropy": 0.2783821925520897, + "epoch": 0.8257638315441783, + "grad_norm": 1.9609375, + "learning_rate": 6.538461538461539e-06, + "loss": 0.28870720863342286, + "mean_token_accuracy": 0.9073394626379013, + "num_tokens": 2333798.0, + "step": 250 + }, + { + "entropy": 0.277759512513876, + "epoch": 0.8587943848059455, + "grad_norm": 1.7734375, + "learning_rate": 6.3553113553113555e-06, + "loss": 0.29060728549957277, + "mean_token_accuracy": 0.9055352889001369, + "num_tokens": 2429910.0, + "step": 260 + }, + { + "entropy": 0.2882726760581136, + "epoch": 0.8918249380677127, + "grad_norm": 1.703125, + "learning_rate": 6.172161172161173e-06, + "loss": 0.2988502740859985, + "mean_token_accuracy": 0.9038613699376583, + "num_tokens": 2524730.0, + "step": 270 + }, + { + "entropy": 0.2855747708119452, + "epoch": 0.9248554913294798, + "grad_norm": 1.90625, + "learning_rate": 5.989010989010989e-06, + "loss": 0.289182186126709, + "mean_token_accuracy": 0.9049803033471108, + "num_tokens": 2615802.0, + "step": 280 + }, + { + "entropy": 0.26923007648438213, + "epoch": 0.9578860445912469, + "grad_norm": 2.84375, + "learning_rate": 5.805860805860807e-06, + "loss": 0.27834036350250246, + "mean_token_accuracy": 0.9109352678060532, + "num_tokens": 2709073.0, + "step": 290 + }, + { + "entropy": 0.2844470454379916, + "epoch": 0.990916597853014, + "grad_norm": 1.640625, + "learning_rate": 5.6227106227106235e-06, + "loss": 0.28873960971832274, + "mean_token_accuracy": 0.9069667845964432, + "num_tokens": 2805135.0, + "step": 300 + }, + { + "epoch": 0.990916597853014, + "eval_entropy": 0.285272175570329, + "eval_loss": 0.28705036640167236, + "eval_mean_token_accuracy": 0.9062359571456909, + "eval_num_tokens": 2805135.0, + "eval_runtime": 17.9038, + "eval_samples_per_second": 30.105, + "eval_steps_per_second": 15.081, + "step": 300 + }, + { + "entropy": 0.2869852078266633, + "epoch": 1.023121387283237, + "grad_norm": 1.9375, + "learning_rate": 5.43956043956044e-06, + "loss": 0.2850792646408081, + "mean_token_accuracy": 0.9064911672702203, + "num_tokens": 2898984.0, + "step": 310 + }, + { + "entropy": 0.2757460119202733, + "epoch": 1.0561519405450042, + "grad_norm": 1.8359375, + "learning_rate": 5.256410256410257e-06, + "loss": 0.2779810905456543, + "mean_token_accuracy": 0.9096895463764667, + "num_tokens": 2991694.0, + "step": 320 + }, + { + "entropy": 0.2600112119689584, + "epoch": 1.0891824938067713, + "grad_norm": 2.0625, + "learning_rate": 5.0732600732600735e-06, + "loss": 0.2662019729614258, + "mean_token_accuracy": 0.912333931028843, + "num_tokens": 3078862.0, + "step": 330 + }, + { + "entropy": 0.27342566605657337, + "epoch": 1.1222130470685383, + "grad_norm": 1.921875, + "learning_rate": 4.890109890109891e-06, + "loss": 0.28096842765808105, + "mean_token_accuracy": 0.9091624312102795, + "num_tokens": 3174256.0, + "step": 340 + }, + { + "entropy": 0.2671937083825469, + "epoch": 1.1552436003303055, + "grad_norm": 1.75, + "learning_rate": 4.706959706959707e-06, + "loss": 0.2723649501800537, + "mean_token_accuracy": 0.9101433955132961, + "num_tokens": 3272477.0, + "step": 350 + }, + { + "entropy": 0.2617026660591364, + "epoch": 1.1882741535920727, + "grad_norm": 1.9453125, + "learning_rate": 4.523809523809524e-06, + "loss": 0.26422107219696045, + "mean_token_accuracy": 0.913913157582283, + "num_tokens": 3364137.0, + "step": 360 + }, + { + "entropy": 0.2595691759139299, + "epoch": 1.2213047068538398, + "grad_norm": 1.5390625, + "learning_rate": 4.340659340659341e-06, + "loss": 0.2714951276779175, + "mean_token_accuracy": 0.911636833101511, + "num_tokens": 3460217.0, + "step": 370 + }, + { + "entropy": 0.2933320404961705, + "epoch": 1.254335260115607, + "grad_norm": 1.9765625, + "learning_rate": 4.157509157509158e-06, + "loss": 0.29573276042938235, + "mean_token_accuracy": 0.902192335575819, + "num_tokens": 3557030.0, + "step": 380 + }, + { + "entropy": 0.2693369995802641, + "epoch": 1.287365813377374, + "grad_norm": 1.890625, + "learning_rate": 3.974358974358974e-06, + "loss": 0.27785425186157225, + "mean_token_accuracy": 0.9090337291359901, + "num_tokens": 3654159.0, + "step": 390 + }, + { + "entropy": 0.2908114368095994, + "epoch": 1.3203963666391412, + "grad_norm": 2.015625, + "learning_rate": 3.7912087912087915e-06, + "loss": 0.30535905361175536, + "mean_token_accuracy": 0.901054035872221, + "num_tokens": 3749664.0, + "step": 400 + }, + { + "epoch": 1.3203963666391412, + "eval_entropy": 0.2783401116176888, + "eval_loss": 0.28495287895202637, + "eval_mean_token_accuracy": 0.9069356251645971, + "eval_num_tokens": 3749664.0, + "eval_runtime": 17.8814, + "eval_samples_per_second": 30.143, + "eval_steps_per_second": 15.099, + "step": 400 + }, + { + "entropy": 0.2749379588291049, + "epoch": 1.3534269199009084, + "grad_norm": 1.84375, + "learning_rate": 3.6080586080586083e-06, + "loss": 0.27795455455780027, + "mean_token_accuracy": 0.9090839587152004, + "num_tokens": 3845841.0, + "step": 410 + }, + { + "entropy": 0.2859474029392004, + "epoch": 1.3864574731626755, + "grad_norm": 1.6796875, + "learning_rate": 3.424908424908425e-06, + "loss": 0.2886682987213135, + "mean_token_accuracy": 0.9059950634837151, + "num_tokens": 3942402.0, + "step": 420 + }, + { + "entropy": 0.2537162031978369, + "epoch": 1.4194880264244425, + "grad_norm": 1.9375, + "learning_rate": 3.2417582417582424e-06, + "loss": 0.25533475875854494, + "mean_token_accuracy": 0.9158934533596039, + "num_tokens": 4034122.0, + "step": 430 + }, + { + "entropy": 0.28255065642297267, + "epoch": 1.4525185796862097, + "grad_norm": 1.7734375, + "learning_rate": 3.058608058608059e-06, + "loss": 0.29817631244659426, + "mean_token_accuracy": 0.9040707617998123, + "num_tokens": 4127068.0, + "step": 440 + }, + { + "entropy": 0.2898306529968977, + "epoch": 1.4855491329479769, + "grad_norm": 1.703125, + "learning_rate": 2.8754578754578755e-06, + "loss": 0.2923931121826172, + "mean_token_accuracy": 0.9030195340514183, + "num_tokens": 4224435.0, + "step": 450 + }, + { + "entropy": 0.27209411058574917, + "epoch": 1.518579686209744, + "grad_norm": 1.7578125, + "learning_rate": 2.6923076923076923e-06, + "loss": 0.28646528720855713, + "mean_token_accuracy": 0.9095881894230843, + "num_tokens": 4321028.0, + "step": 460 + }, + { + "entropy": 0.280727200768888, + "epoch": 1.5516102394715112, + "grad_norm": 1.7421875, + "learning_rate": 2.509157509157509e-06, + "loss": 0.2891413688659668, + "mean_token_accuracy": 0.9067636162042618, + "num_tokens": 4414916.0, + "step": 470 + }, + { + "entropy": 0.27109829243272543, + "epoch": 1.5846407927332784, + "grad_norm": 1.8359375, + "learning_rate": 2.3260073260073264e-06, + "loss": 0.26625516414642336, + "mean_token_accuracy": 0.9099020563066006, + "num_tokens": 4505708.0, + "step": 480 + }, + { + "entropy": 0.2729395367205143, + "epoch": 1.6176713459950454, + "grad_norm": 1.7890625, + "learning_rate": 2.1428571428571427e-06, + "loss": 0.2779061317443848, + "mean_token_accuracy": 0.909747340530157, + "num_tokens": 4600244.0, + "step": 490 + }, + { + "entropy": 0.2682301852852106, + "epoch": 1.6507018992568125, + "grad_norm": 1.6953125, + "learning_rate": 1.95970695970696e-06, + "loss": 0.274446964263916, + "mean_token_accuracy": 0.9094639919698239, + "num_tokens": 4688147.0, + "step": 500 + }, + { + "epoch": 1.6507018992568125, + "eval_entropy": 0.2756172451708052, + "eval_loss": 0.2841346263885498, + "eval_mean_token_accuracy": 0.9072470570052111, + "eval_num_tokens": 4688147.0, + "eval_runtime": 17.9119, + "eval_samples_per_second": 30.092, + "eval_steps_per_second": 15.074, + "step": 500 + }, + { + "entropy": 0.26342453975230457, + "epoch": 1.6837324525185797, + "grad_norm": 1.890625, + "learning_rate": 1.7765567765567768e-06, + "loss": 0.26432271003723146, + "mean_token_accuracy": 0.9114409819245338, + "num_tokens": 4780124.0, + "step": 510 + }, + { + "entropy": 0.28657868495211003, + "epoch": 1.7167630057803467, + "grad_norm": 1.7734375, + "learning_rate": 1.5934065934065933e-06, + "loss": 0.2983593225479126, + "mean_token_accuracy": 0.9028328903019428, + "num_tokens": 4874633.0, + "step": 520 + }, + { + "entropy": 0.25633321776986123, + "epoch": 1.7497935590421139, + "grad_norm": 1.8359375, + "learning_rate": 1.4102564102564104e-06, + "loss": 0.2606488227844238, + "mean_token_accuracy": 0.9153999648988247, + "num_tokens": 4967263.0, + "step": 530 + }, + { + "entropy": 0.26120030507445335, + "epoch": 1.782824112303881, + "grad_norm": 2.078125, + "learning_rate": 1.2271062271062271e-06, + "loss": 0.2706313610076904, + "mean_token_accuracy": 0.9130682416260243, + "num_tokens": 5056736.0, + "step": 540 + }, + { + "entropy": 0.27009565513581035, + "epoch": 1.8158546655656482, + "grad_norm": 1.8828125, + "learning_rate": 1.0439560439560442e-06, + "loss": 0.277858567237854, + "mean_token_accuracy": 0.9087999157607556, + "num_tokens": 5150617.0, + "step": 550 + }, + { + "entropy": 0.2819518899545074, + "epoch": 1.8488852188274154, + "grad_norm": 1.9609375, + "learning_rate": 8.608058608058609e-07, + "loss": 0.28743035793304444, + "mean_token_accuracy": 0.9067244723439216, + "num_tokens": 5243538.0, + "step": 560 + }, + { + "entropy": 0.2858045116066933, + "epoch": 1.8819157720891826, + "grad_norm": 2.203125, + "learning_rate": 6.776556776556777e-07, + "loss": 0.2939825296401978, + "mean_token_accuracy": 0.904579871147871, + "num_tokens": 5331441.0, + "step": 570 + }, + { + "entropy": 0.2926233192905784, + "epoch": 1.9149463253509498, + "grad_norm": 2.3125, + "learning_rate": 4.945054945054946e-07, + "loss": 0.29961700439453126, + "mean_token_accuracy": 0.9033655568957328, + "num_tokens": 5426734.0, + "step": 580 + }, + { + "entropy": 0.2706678254529834, + "epoch": 1.9479768786127167, + "grad_norm": 2.03125, + "learning_rate": 3.113553113553114e-07, + "loss": 0.27953526973724363, + "mean_token_accuracy": 0.9093762949109078, + "num_tokens": 5518190.0, + "step": 590 + }, + { + "entropy": 0.27056904807686805, + "epoch": 1.981007431874484, + "grad_norm": 2.09375, + "learning_rate": 1.282051282051282e-07, + "loss": 0.2764536619186401, + "mean_token_accuracy": 0.9110228553414345, + "num_tokens": 5609238.0, + "step": 600 + }, + { + "epoch": 1.981007431874484, + "eval_entropy": 0.2757842331572815, + "eval_loss": 0.28394654393196106, + "eval_mean_token_accuracy": 0.9073928307603907, + "eval_num_tokens": 5609238.0, + "eval_runtime": 18.1138, + "eval_samples_per_second": 29.756, + "eval_steps_per_second": 14.906, + "step": 600 + }, + { + "epoch": 2.0, + "eval_entropy": 0.2757109238593667, + "eval_loss": 0.2840007245540619, + "eval_mean_token_accuracy": 0.9072579809912928, + "eval_num_tokens": 5663138.0, + "eval_runtime": 18.1948, + "eval_samples_per_second": 29.624, + "eval_steps_per_second": 14.839, + "step": 606 + } + ], + "logging_steps": 10, + "max_steps": 606, + "num_input_tokens_seen": 0, + "num_train_epochs": 2, + "save_steps": 100, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": true + }, + "attributes": {} + } + }, + "total_flos": 6.072645019326874e+16, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-606/training_args.bin b/checkpoint-606/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..133cb6661b0124bb9db0d7d208d748fd736ea689 --- /dev/null +++ b/checkpoint-606/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6696e1fef8bde5e22de2f5ae18c284f2b2d35a345157db5151571518385fb38b +size 5713 diff --git a/checkpoint-674/chat_template.jinja b/checkpoint-674/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..01be9b307daa2d425f7c168c9fb145a286e0afb4 --- /dev/null +++ b/checkpoint-674/chat_template.jinja @@ -0,0 +1,89 @@ +{%- if tools %} + {{- '<|im_start|>system\n' }} + {%- if messages[0].role == 'system' %} + {{- messages[0].content + '\n\n' }} + {%- endif %} + {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within XML tags:\n" }} + {%- for tool in tools %} + {{- "\n" }} + {{- tool | tojson }} + {%- endfor %} + {{- "\n\n\nFor each function call, return a json object with function name and arguments within XML tags:\n\n{\"name\": , \"arguments\": }\n<|im_end|>\n" }} +{%- else %} + {%- if messages[0].role == 'system' %} + {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %} +{%- for message in messages[::-1] %} + {%- set index = (messages|length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} +{%- endfor %} +{%- for message in messages %} + {%- if message.content is string %} + {%- set content = message.content %} + {%- else %} + {%- set content = '' %} + {%- endif %} + {%- if (message.role == "user") or (message.role == "system" and not loop.first) %} + {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }} + {%- elif message.role == "assistant" %} + {%- set reasoning_content = '' %} + {%- if message.reasoning_content is string %} + {%- set reasoning_content = message.reasoning_content %} + {%- else %} + {%- if '' in content %} + {%- set reasoning_content = content.split('')[0].rstrip('\n').split('')[-1].lstrip('\n') %} + {%- set content = content.split('')[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- if loop.index0 > ns.last_query_index %} + {%- if loop.last or (not loop.last and reasoning_content) %} + {{- '<|im_start|>' + message.role + '\n\n' + reasoning_content.strip('\n') + '\n\n\n' + content.lstrip('\n') }} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- else %} + {{- '<|im_start|>' + message.role + '\n' + content }} + {%- endif %} + {%- if message.tool_calls %} + {%- for tool_call in message.tool_calls %} + {%- if (loop.first and content) or (not loop.first) %} + {{- '\n' }} + {%- endif %} + {%- if tool_call.function %} + {%- set tool_call = tool_call.function %} + {%- endif %} + {{- '\n{"name": "' }} + {{- tool_call.name }} + {{- '", "arguments": ' }} + {%- if tool_call.arguments is string %} + {{- tool_call.arguments }} + {%- else %} + {{- tool_call.arguments | tojson }} + {%- endif %} + {{- '}\n' }} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == "tool" %} + {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %} + {{- '<|im_start|>user' }} + {%- endif %} + {{- '\n\n' }} + {{- content }} + {{- '\n' }} + {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if enable_thinking is defined and enable_thinking is false %} + {{- '\n\n\n\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/checkpoint-674/config.json b/checkpoint-674/config.json new file mode 100644 index 0000000000000000000000000000000000000000..1d219d6a6d288625229a71646710b85cf7ae5def --- /dev/null +++ b/checkpoint-674/config.json @@ -0,0 +1,63 @@ +{ + "architectures": [ + "Qwen3ForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": null, + "dtype": "bfloat16", + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 2048, + "initializer_range": 0.02, + "intermediate_size": 6144, + "layer_types": [ + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention", + "full_attention" + ], + "max_position_embeddings": 40960, + "max_window_layers": 28, + "model_type": "qwen3", + "num_attention_heads": 16, + "num_hidden_layers": 28, + "num_key_value_heads": 8, + "pad_token_id": 151643, + "rms_norm_eps": 1e-06, + "rope_parameters": { + "rope_theta": 1000000, + "rope_type": "default" + }, + "sliding_window": null, + "tie_word_embeddings": true, + "transformers_version": "5.7.0", + "use_cache": false, + "use_sliding_window": false, + "vocab_size": 151936 +} diff --git a/checkpoint-674/generation_config.json b/checkpoint-674/generation_config.json new file mode 100644 index 0000000000000000000000000000000000000000..5ec133d867b1427d43a5ee0ea332bd088fc12013 --- /dev/null +++ b/checkpoint-674/generation_config.json @@ -0,0 +1,12 @@ +{ + "do_sample": true, + "eos_token_id": [ + 151645, + 151643 + ], + "pad_token_id": 151643, + "temperature": 0.6, + "top_k": 20, + "top_p": 0.95, + "transformers_version": "5.7.0" +} diff --git a/checkpoint-674/model.safetensors b/checkpoint-674/model.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..cc885a501a0fffef97b843b5248e3bb7cfe46172 --- /dev/null +++ b/checkpoint-674/model.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3af4557502ae7c7853ed3b684aa50cfd00317aafc237f2dda688075bb3dc81c1 +size 3441185608 diff --git a/checkpoint-674/optimizer.pt b/checkpoint-674/optimizer.pt new file mode 100644 index 0000000000000000000000000000000000000000..4f8c22d5c745517a99843c614c05da8a366a7d78 --- /dev/null +++ b/checkpoint-674/optimizer.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9ca0ddbeff189519bd91d1474687c97eb42f33e1dc183a5769974dadc5e0cd7b +size 6882572207 diff --git a/checkpoint-674/rng_state.pth b/checkpoint-674/rng_state.pth new file mode 100644 index 0000000000000000000000000000000000000000..1feba1a6538e93b94696d3773853dbc8947b0cad --- /dev/null +++ b/checkpoint-674/rng_state.pth @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:098b29492211804ab324a36f37466821d948280bb74fce4ba895c03f13ecd878 +size 14645 diff --git a/checkpoint-674/scheduler.pt b/checkpoint-674/scheduler.pt new file mode 100644 index 0000000000000000000000000000000000000000..52e04de5969d9a926e44ed315d79199b8ca27698 --- /dev/null +++ b/checkpoint-674/scheduler.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:01dbf241e935857d2a5486058c590aafb795b79d6ef3f0dc038c0d85e2187d7d +size 1465 diff --git a/checkpoint-674/tokenizer.json b/checkpoint-674/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..c7afbed2efcdf019f88ab0572ec29d3bf595dfe2 --- /dev/null +++ b/checkpoint-674/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506 +size 11422650 diff --git a/checkpoint-674/tokenizer_config.json b/checkpoint-674/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..770e41d6c92519d525eede4cbcf3ba27f6425311 --- /dev/null +++ b/checkpoint-674/tokenizer_config.json @@ -0,0 +1,30 @@ +{ + "add_prefix_space": false, + "backend": "tokenizers", + "bos_token": null, + "clean_up_tokenization_spaces": false, + "eos_token": "<|im_end|>", + "errors": "replace", + "extra_special_tokens": [ + "<|im_start|>", + "<|im_end|>", + "<|object_ref_start|>", + "<|object_ref_end|>", + "<|box_start|>", + "<|box_end|>", + "<|quad_start|>", + "<|quad_end|>", + "<|vision_start|>", + "<|vision_end|>", + "<|vision_pad|>", + "<|image_pad|>", + "<|video_pad|>" + ], + "is_local": false, + "local_files_only": false, + "model_max_length": 131072, + "pad_token": "<|endoftext|>", + "split_special_tokens": false, + "tokenizer_class": "Qwen2Tokenizer", + "unk_token": null +} diff --git a/checkpoint-674/trainer_state.json b/checkpoint-674/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..54b5d973ad7d8fb6a649dbc5f72a0c5653d31516 --- /dev/null +++ b/checkpoint-674/trainer_state.json @@ -0,0 +1,704 @@ +{ + "best_global_step": null, + "best_metric": null, + "best_model_checkpoint": null, + "epoch": 2.0, + "eval_steps": 500, + "global_step": 674, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "entropy": NaN, + "epoch": 0.029717682020802376, + "grad_norm": NaN, + "learning_rate": 1.8000000000000001e-06, + "loss": 1005514863411.2, + "mean_token_accuracy": 0.07770282882993343, + "num_tokens": 97764.0, + "step": 10 + }, + { + "entropy": NaN, + "epoch": 0.05943536404160475, + "grad_norm": NaN, + "learning_rate": 3.8000000000000005e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007392044077278115, + "num_tokens": 190372.0, + "step": 20 + }, + { + "entropy": NaN, + "epoch": 0.08915304606240713, + "grad_norm": NaN, + "learning_rate": 5.8e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000383764869911829, + "num_tokens": 280928.0, + "step": 30 + }, + { + "entropy": NaN, + "epoch": 0.1188707280832095, + "grad_norm": NaN, + "learning_rate": 7.800000000000002e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0003212536917999387, + "num_tokens": 380644.0, + "step": 40 + }, + { + "entropy": NaN, + "epoch": 0.1485884101040119, + "grad_norm": NaN, + "learning_rate": 9.800000000000001e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0003853117574180942, + "num_tokens": 475467.0, + "step": 50 + }, + { + "entropy": NaN, + "epoch": 0.17830609212481427, + "grad_norm": NaN, + "learning_rate": 1.18e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0003134153150313068, + "num_tokens": 572411.0, + "step": 60 + }, + { + "entropy": NaN, + "epoch": 0.20802377414561665, + "grad_norm": NaN, + "learning_rate": 1.38e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011001345737895463, + "num_tokens": 661517.0, + "step": 70 + }, + { + "entropy": NaN, + "epoch": 0.237741456166419, + "grad_norm": NaN, + "learning_rate": 1.58e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012231820757733658, + "num_tokens": 752142.0, + "step": 80 + }, + { + "entropy": NaN, + "epoch": 0.2674591381872214, + "grad_norm": NaN, + "learning_rate": 1.7800000000000002e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011551132018212229, + "num_tokens": 850399.0, + "step": 90 + }, + { + "entropy": NaN, + "epoch": 0.2971768202080238, + "grad_norm": NaN, + "learning_rate": 1.98e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00047943509634933433, + "num_tokens": 954759.0, + "step": 100 + }, + { + "entropy": NaN, + "epoch": 0.32689450222882616, + "grad_norm": NaN, + "learning_rate": 1.9802414928649837e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008438902725174558, + "num_tokens": 1050439.0, + "step": 110 + }, + { + "entropy": NaN, + "epoch": 0.35661218424962854, + "grad_norm": NaN, + "learning_rate": 1.958287596048299e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009903057703922967, + "num_tokens": 1135089.0, + "step": 120 + }, + { + "entropy": NaN, + "epoch": 0.3863298662704309, + "grad_norm": NaN, + "learning_rate": 1.9363336992316137e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00040032515971688554, + "num_tokens": 1228149.0, + "step": 130 + }, + { + "entropy": NaN, + "epoch": 0.4160475482912333, + "grad_norm": NaN, + "learning_rate": 1.9143798024149288e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0010247966441966128, + "num_tokens": 1331672.0, + "step": 140 + }, + { + "entropy": NaN, + "epoch": 0.4457652303120357, + "grad_norm": NaN, + "learning_rate": 1.892425905598244e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000843161555530969, + "num_tokens": 1421314.0, + "step": 150 + }, + { + "entropy": NaN, + "epoch": 0.475482912332838, + "grad_norm": NaN, + "learning_rate": 1.8704720087815587e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007889680869993754, + "num_tokens": 1511701.0, + "step": 160 + }, + { + "entropy": NaN, + "epoch": 0.5052005943536404, + "grad_norm": NaN, + "learning_rate": 1.848518111964874e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007493881763366517, + "num_tokens": 1605063.0, + "step": 170 + }, + { + "entropy": NaN, + "epoch": 0.5349182763744428, + "grad_norm": NaN, + "learning_rate": 1.826564215148189e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000567350817436818, + "num_tokens": 1703107.0, + "step": 180 + }, + { + "entropy": NaN, + "epoch": 0.5646359583952452, + "grad_norm": NaN, + "learning_rate": 1.804610318331504e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00027060438151238486, + "num_tokens": 1796279.0, + "step": 190 + }, + { + "entropy": NaN, + "epoch": 0.5943536404160475, + "grad_norm": NaN, + "learning_rate": 1.782656421514819e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008226798396208324, + "num_tokens": 1893647.0, + "step": 200 + }, + { + "entropy": NaN, + "epoch": 0.6240713224368499, + "grad_norm": NaN, + "learning_rate": 1.760702524698134e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007338574505411088, + "num_tokens": 1982325.0, + "step": 210 + }, + { + "entropy": NaN, + "epoch": 0.6537890044576523, + "grad_norm": NaN, + "learning_rate": 1.7387486278814492e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004891513846814633, + "num_tokens": 2071869.0, + "step": 220 + }, + { + "entropy": NaN, + "epoch": 0.6835066864784547, + "grad_norm": NaN, + "learning_rate": 1.716794731064764e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00038509477453771976, + "num_tokens": 2168370.0, + "step": 230 + }, + { + "entropy": NaN, + "epoch": 0.7132243684992571, + "grad_norm": NaN, + "learning_rate": 1.694840834248079e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009255842014681548, + "num_tokens": 2263211.0, + "step": 240 + }, + { + "entropy": NaN, + "epoch": 0.7429420505200595, + "grad_norm": NaN, + "learning_rate": 1.6728869374313943e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011857455348945223, + "num_tokens": 2352425.0, + "step": 250 + }, + { + "entropy": NaN, + "epoch": 0.7726597325408618, + "grad_norm": NaN, + "learning_rate": 1.6509330406147094e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0008204675337765366, + "num_tokens": 2445217.0, + "step": 260 + }, + { + "entropy": NaN, + "epoch": 0.8023774145616642, + "grad_norm": NaN, + "learning_rate": 1.6289791437980242e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0013553448639868292, + "num_tokens": 2537677.0, + "step": 270 + }, + { + "entropy": NaN, + "epoch": 0.8320950965824666, + "grad_norm": NaN, + "learning_rate": 1.6070252469813393e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011630529144895263, + "num_tokens": 2631646.0, + "step": 280 + }, + { + "entropy": NaN, + "epoch": 0.861812778603269, + "grad_norm": NaN, + "learning_rate": 1.5850713501646545e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009409735575900413, + "num_tokens": 2722632.0, + "step": 290 + }, + { + "entropy": NaN, + "epoch": 0.8915304606240714, + "grad_norm": NaN, + "learning_rate": 1.5631174533479693e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006484775040007662, + "num_tokens": 2819185.0, + "step": 300 + }, + { + "entropy": NaN, + "epoch": 0.9212481426448736, + "grad_norm": NaN, + "learning_rate": 1.5411635565312844e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011682231881422922, + "num_tokens": 2910483.0, + "step": 310 + }, + { + "entropy": NaN, + "epoch": 0.950965824665676, + "grad_norm": NaN, + "learning_rate": 1.5192096597145995e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004708930806373246, + "num_tokens": 2997057.0, + "step": 320 + }, + { + "entropy": NaN, + "epoch": 0.9806835066864784, + "grad_norm": NaN, + "learning_rate": 1.4972557628979145e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011955970534472727, + "num_tokens": 3091284.0, + "step": 330 + }, + { + "entropy": NaN, + "epoch": 1.0089153046062407, + "grad_norm": NaN, + "learning_rate": 1.4753018660812295e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006843029964090276, + "num_tokens": 3177800.0, + "step": 340 + }, + { + "entropy": NaN, + "epoch": 1.0386329866270432, + "grad_norm": NaN, + "learning_rate": 1.4533479692645444e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0005406756106822285, + "num_tokens": 3271989.0, + "step": 350 + }, + { + "entropy": NaN, + "epoch": 1.0683506686478454, + "grad_norm": NaN, + "learning_rate": 1.4313940724478596e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0007009803419350646, + "num_tokens": 3365082.0, + "step": 360 + }, + { + "entropy": NaN, + "epoch": 1.098068350668648, + "grad_norm": NaN, + "learning_rate": 1.4094401756311746e-05, + "loss": 0.0, + "mean_token_accuracy": 0.001003894711175235, + "num_tokens": 3465865.0, + "step": 370 + }, + { + "entropy": NaN, + "epoch": 1.1277860326894502, + "grad_norm": NaN, + "learning_rate": 1.3874862788144897e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004136150790145621, + "num_tokens": 3564998.0, + "step": 380 + }, + { + "entropy": NaN, + "epoch": 1.1575037147102525, + "grad_norm": NaN, + "learning_rate": 1.3655323819978048e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000998112405068241, + "num_tokens": 3661508.0, + "step": 390 + }, + { + "entropy": NaN, + "epoch": 1.187221396731055, + "grad_norm": NaN, + "learning_rate": 1.3435784851811198e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0009621107514249161, + "num_tokens": 3755395.0, + "step": 400 + }, + { + "entropy": NaN, + "epoch": 1.2169390787518575, + "grad_norm": NaN, + "learning_rate": 1.3216245883644348e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004976072828867473, + "num_tokens": 3842994.0, + "step": 410 + }, + { + "entropy": NaN, + "epoch": 1.2466567607726597, + "grad_norm": NaN, + "learning_rate": 1.2996706915477499e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0003434549740632065, + "num_tokens": 3934618.0, + "step": 420 + }, + { + "entropy": NaN, + "epoch": 1.276374442793462, + "grad_norm": NaN, + "learning_rate": 1.2777167947310649e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00026364591685705817, + "num_tokens": 4026711.0, + "step": 430 + }, + { + "entropy": NaN, + "epoch": 1.3060921248142645, + "grad_norm": NaN, + "learning_rate": 1.2557628979143798e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0006349188319290988, + "num_tokens": 4114609.0, + "step": 440 + }, + { + "entropy": NaN, + "epoch": 1.3358098068350668, + "grad_norm": NaN, + "learning_rate": 1.2338090010976948e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004265022376785055, + "num_tokens": 4203953.0, + "step": 450 + }, + { + "entropy": NaN, + "epoch": 1.3655274888558693, + "grad_norm": NaN, + "learning_rate": 1.2118551042810101e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012633154692593962, + "num_tokens": 4298319.0, + "step": 460 + }, + { + "entropy": NaN, + "epoch": 1.3952451708766715, + "grad_norm": NaN, + "learning_rate": 1.189901207464325e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0011256658195634372, + "num_tokens": 4396020.0, + "step": 470 + }, + { + "entropy": NaN, + "epoch": 1.424962852897474, + "grad_norm": NaN, + "learning_rate": 1.16794731064764e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0013131745137798135, + "num_tokens": 4491203.0, + "step": 480 + }, + { + "entropy": NaN, + "epoch": 1.4546805349182763, + "grad_norm": NaN, + "learning_rate": 1.1459934138309552e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012716218378045596, + "num_tokens": 4582199.0, + "step": 490 + }, + { + "entropy": NaN, + "epoch": 1.4843982169390788, + "grad_norm": NaN, + "learning_rate": 1.1240395170142701e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0014869970778818241, + "num_tokens": 4672589.0, + "step": 500 + }, + { + "entropy": NaN, + "epoch": 1.514115898959881, + "grad_norm": NaN, + "learning_rate": 1.1020856201975851e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00046981508567114363, + "num_tokens": 4767693.0, + "step": 510 + }, + { + "entropy": NaN, + "epoch": 1.5438335809806834, + "grad_norm": NaN, + "learning_rate": 1.0801317233809e-05, + "loss": 0.0, + "mean_token_accuracy": 0.000601445968641201, + "num_tokens": 4860371.0, + "step": 520 + }, + { + "entropy": NaN, + "epoch": 1.5735512630014858, + "grad_norm": NaN, + "learning_rate": 1.0581778265642154e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0012073652920662425, + "num_tokens": 4950622.0, + "step": 530 + }, + { + "entropy": NaN, + "epoch": 1.6032689450222883, + "grad_norm": NaN, + "learning_rate": 1.0362239297475303e-05, + "loss": 0.0, + "mean_token_accuracy": 0.00028707647215924225, + "num_tokens": 5044878.0, + "step": 540 + }, + { + "entropy": NaN, + "epoch": 1.6329866270430906, + "grad_norm": NaN, + "learning_rate": 1.0142700329308453e-05, + "loss": 0.0, + "mean_token_accuracy": 0.0004101752907445189, + "num_tokens": 5140938.0, + "step": 550 + }, + { + "entropy": NaN, + "epoch": 1.6627043090638929, + "grad_norm": NaN, + "learning_rate": 9.923161361141603e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008072987395280507, + "num_tokens": 5232852.0, + "step": 560 + }, + { + "entropy": NaN, + "epoch": 1.6924219910846954, + "grad_norm": NaN, + "learning_rate": 9.703622392974754e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0012222853780258446, + "num_tokens": 5321835.0, + "step": 570 + }, + { + "entropy": NaN, + "epoch": 1.7221396731054979, + "grad_norm": NaN, + "learning_rate": 9.484083424807904e-06, + "loss": 0.0, + "mean_token_accuracy": 0.001064640874392353, + "num_tokens": 5416464.0, + "step": 580 + }, + { + "entropy": NaN, + "epoch": 1.7518573551263001, + "grad_norm": NaN, + "learning_rate": 9.264544456641055e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0009612867230316624, + "num_tokens": 5515169.0, + "step": 590 + }, + { + "entropy": NaN, + "epoch": 1.7815750371471024, + "grad_norm": NaN, + "learning_rate": 9.045005488474205e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008774219051701948, + "num_tokens": 5611971.0, + "step": 600 + }, + { + "entropy": NaN, + "epoch": 1.811292719167905, + "grad_norm": NaN, + "learning_rate": 8.825466520307355e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007863394406740553, + "num_tokens": 5704255.0, + "step": 610 + }, + { + "entropy": NaN, + "epoch": 1.8410104011887074, + "grad_norm": NaN, + "learning_rate": 8.605927552140506e-06, + "loss": 0.0, + "mean_token_accuracy": 0.000578320066415472, + "num_tokens": 5800440.0, + "step": 620 + }, + { + "entropy": NaN, + "epoch": 1.8707280832095097, + "grad_norm": NaN, + "learning_rate": 8.386388583973656e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0004275172817870043, + "num_tokens": 5890898.0, + "step": 630 + }, + { + "entropy": NaN, + "epoch": 1.900445765230312, + "grad_norm": NaN, + "learning_rate": 8.166849615806807e-06, + "loss": 0.0, + "mean_token_accuracy": 0.001425291335908696, + "num_tokens": 5987228.0, + "step": 640 + }, + { + "entropy": NaN, + "epoch": 1.9301634472511144, + "grad_norm": NaN, + "learning_rate": 7.947310647639957e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0006059414023184218, + "num_tokens": 6076579.0, + "step": 650 + }, + { + "entropy": NaN, + "epoch": 1.959881129271917, + "grad_norm": NaN, + "learning_rate": 7.727771679473108e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0008722793892957271, + "num_tokens": 6171013.0, + "step": 660 + }, + { + "entropy": NaN, + "epoch": 1.9895988112927192, + "grad_norm": NaN, + "learning_rate": 7.508232711306257e-06, + "loss": 0.0, + "mean_token_accuracy": 0.0007748565993097145, + "num_tokens": 6265432.0, + "step": 670 + } + ], + "logging_steps": 10, + "max_steps": 1011, + "num_input_tokens_seen": 0, + "num_train_epochs": 3, + "save_steps": 500, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": false + }, + "attributes": {} + } + }, + "total_flos": 6.753063653143757e+16, + "train_batch_size": 2, + "trial_name": null, + "trial_params": null +} diff --git a/checkpoint-674/training_args.bin b/checkpoint-674/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..737a87caa4fc2f704a1fe56a508ca4d335ca783d --- /dev/null +++ b/checkpoint-674/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ae0f7084660199ed7fd16027c0c6af5e726790f32a022b019f1277bec4e01142 +size 5713 diff --git a/config.json b/config.json index cd2e0cfcc25e9201303ab4390eee904c9cc5a285..c849622762e06273c720e07098ab07dbbea3b8b0 100644 --- a/config.json +++ b/config.json @@ -5,7 +5,7 @@ "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": null, - "dtype": "bfloat16", + "dtype": "float32", "eos_token_id": 151645, "head_dim": 128, "hidden_act": "silu", diff --git a/eval_results.diverse.json b/eval_results.diverse.json new file mode 100644 index 0000000000000000000000000000000000000000..1f6ec55ddc2f6134480996e20074310f889d6b16 --- /dev/null +++ b/eval_results.diverse.json @@ -0,0 +1,207 @@ +{ + "checkpoint": "/scratch/checkpoints/exp6_sft_numinamath_dpo", + "is_local": true, + "eval_tier": "diverse", + "sampling": { + "n": 8, + "temperature": 0.6, + "top_p": 0.95, + "top_k": 20, + "max_tokens": 3500 + }, + "benchmarks": { + "numinamath_benchmark": { + "source": "numinamath_benchmark", + "overall": { + "n_problems": 500, + "n_completions": 8, + "pass@1": 0.3795, + "pass@8": 0.612 + }, + "by_source": { + "numina:amc_aime": { + "n_problems": 1, + "n_completions": 8, + "pass@1": 0.25, + "pass@8": 1.0 + }, + "numina:aops_forum": { + "n_problems": 9, + "n_completions": 8, + "pass@1": 0.08333333333333333, + "pass@8": 0.4444444444444444 + }, + "numina:cn_contest": { + "n_problems": 7, + "n_completions": 8, + "pass@1": 0.0, + "pass@8": 0.0 + }, + "numina:cn_k12": { + "n_problems": 200, + "n_completions": 8, + "pass@1": 0.331875, + "pass@8": 0.62 + }, + "numina:inequalities": { + "n_problems": 1, + "n_completions": 8, + "pass@1": 0.0, + "pass@8": 0.0 + }, + "numina:metamath": { + "n_problems": 7, + "n_completions": 8, + "pass@1": 0.1964285714285714, + "pass@8": 0.7142857142857143 + }, + "numina:olympiads": { + "n_problems": 49, + "n_completions": 8, + "pass@1": 0.06887755102040816, + "pass@8": 0.20408163265306123 + }, + "numina:olympiads_ref": { + "n_problems": 1, + "n_completions": 8, + "pass@1": 0.25, + "pass@8": 1.0 + }, + "numina:orca_math": { + "n_problems": 112, + "n_completions": 8, + "pass@1": 0.6149553571428571, + "pass@8": 0.8125 + }, + "numina:synthetic_math": { + "n_problems": 113, + "n_completions": 8, + "pass@1": 0.42920353982300885, + "pass@8": 0.6194690265486725 + } + }, + "by_topic": { + "Algebra": { + "n_problems": 277, + "n_completions": 8, + "pass@1": 0.4453971119133574, + "pass@8": 0.6895306859205776 + }, + "Calculus": { + "n_problems": 17, + "n_completions": 8, + "pass@1": 0.20588235294117646, + "pass@8": 0.35294117647058826 + }, + "Combinatorics": { + "n_problems": 31, + "n_completions": 8, + "pass@1": 0.2862903225806452, + "pass@8": 0.5806451612903226 + }, + "Geometry": { + "n_problems": 84, + "n_completions": 8, + "pass@1": 0.3169642857142857, + "pass@8": 0.5357142857142857 + }, + "Inequalities": { + "n_problems": 11, + "n_completions": 8, + "pass@1": 0.375, + "pass@8": 0.6363636363636364 + }, + "Logic and Puzzles": { + "n_problems": 20, + "n_completions": 8, + "pass@1": 0.45, + "pass@8": 0.65 + }, + "Number Theory": { + "n_problems": 54, + "n_completions": 8, + "pass@1": 0.24537037037037038, + "pass@8": 0.46296296296296297 + }, + "Other": { + "n_problems": 6, + "n_completions": 8, + "pass@1": 0.16666666666666666, + "pass@8": 0.16666666666666666 + } + } + }, + "diverse_math_benchmark": { + "source": "diverse_math_benchmark", + "overall": { + "n_problems": 150, + "n_completions": 8, + "pass@1": 0.3275, + "pass@8": 0.5466666666666666 + }, + "by_source": { + "aime_2024": { + "n_problems": 17, + "n_completions": 8, + "pass@1": 0.036764705882352935, + "pass@8": 0.11764705882352941 + }, + "aime_2025": { + "n_problems": 13, + "n_completions": 8, + "pass@1": 0.08653846153846154, + "pass@8": 0.15384615384615385 + }, + "math_hard": { + "n_problems": 50, + "n_completions": 8, + "pass@1": 0.495, + "pass@8": 0.78 + }, + "numina:amc_aime": { + "n_problems": 1, + "n_completions": 8, + "pass@1": 0.125, + "pass@8": 1.0 + }, + "numina:aops_forum": { + "n_problems": 2, + "n_completions": 8, + "pass@1": 0.0, + "pass@8": 0.0 + }, + "numina:cn_k12": { + "n_problems": 26, + "n_completions": 8, + "pass@1": 0.32211538461538464, + "pass@8": 0.5769230769230769 + }, + "numina:metamath": { + "n_problems": 1, + "n_completions": 8, + "pass@1": 0.0, + "pass@8": 0.0 + }, + "numina:olympiads": { + "n_problems": 17, + "n_completions": 8, + "pass@1": 0.08088235294117647, + "pass@8": 0.29411764705882354 + }, + "numina:orca_math": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.6964285714285714, + "pass@8": 0.9285714285714286 + }, + "numina:synthetic_math": { + "n_problems": 9, + "n_completions": 8, + "pass@1": 0.3333333333333333, + "pass@8": 0.5555555555555556 + } + } + } + }, + "note": "All benchmarks use identical sampling params so numbers compare directly across experiments (baseline, SFT v1/v2, GRPO). Hard benchmarks (math_hard_100, aime_benchmark) are decontaminated against MixLong (the SFT pool)." +} \ No newline at end of file diff --git a/eval_results.json b/eval_results.json new file mode 100644 index 0000000000000000000000000000000000000000..34bf083a93ff87791f872d45ce86fa8c43f28737 --- /dev/null +++ b/eval_results.json @@ -0,0 +1,114 @@ +{ + "checkpoint": "/scratch/checkpoints/exp6_sft_numinamath_dpo", + "is_local": true, + "eval_tier": "fast", + "sampling": { + "n": 8, + "temperature": 0.3, + "top_p": 0.95, + "top_k": 20, + "max_tokens": 3500 + }, + "benchmarks": { + "ci_gate_math": { + "source": "ci_gate_math", + "overall": { + "n_problems": 10, + "n_completions": 8, + "pass@1": 0.2, + "pass@8": 0.3 + } + }, + "math_hard_100": { + "source": "math_hard_100", + "overall": { + "n_problems": 100, + "n_completions": 8, + "pass@1": 0.49375, + "pass@8": 0.75 + }, + "by_level": { + "4": { + "n_problems": 51, + "n_completions": 8, + "pass@1": 0.6029411764705882, + "pass@8": 0.8431372549019608 + }, + "5": { + "n_problems": 49, + "n_completions": 8, + "pass@1": 0.38010204081632654, + "pass@8": 0.6530612244897959 + } + }, + "by_topic": { + "algebra": { + "n_problems": 15, + "n_completions": 8, + "pass@1": 0.7916666666666666, + "pass@8": 0.9333333333333333 + }, + "counting_and_probability": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.2857142857142857, + "pass@8": 0.5 + }, + "geometry": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.3928571428571428, + "pass@8": 0.7142857142857143 + }, + "intermediate_algebra": { + "n_problems": 15, + "n_completions": 8, + "pass@1": 0.4166666666666667, + "pass@8": 0.6666666666666666 + }, + "number_theory": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.5178571428571429, + "pass@8": 0.7857142857142857 + }, + "prealgebra": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.6607142857142857, + "pass@8": 1.0 + }, + "precalculus": { + "n_problems": 14, + "n_completions": 8, + "pass@1": 0.375, + "pass@8": 0.6428571428571429 + } + } + }, + "aime_benchmark": { + "source": "aime_benchmark", + "overall": { + "n_problems": 60, + "n_completions": 8, + "pass@1": 0.031249999999999993, + "pass@8": 0.11666666666666667 + }, + "by_source": { + "aime_2024": { + "n_problems": 30, + "n_completions": 8, + "pass@1": 0.029166666666666664, + "pass@8": 0.13333333333333333 + }, + "aime_2025": { + "n_problems": 30, + "n_completions": 8, + "pass@1": 0.033333333333333326, + "pass@8": 0.1 + } + } + } + }, + "note": "All benchmarks use identical sampling params so numbers compare directly across experiments (baseline, SFT v1/v2, GRPO). Hard benchmarks (math_hard_100, aime_benchmark) are decontaminated against MixLong (the SFT pool)." +} \ No newline at end of file diff --git a/generation_config.json b/generation_config.json index 9526fbb9cfca6c6b8911fa4f1995a5c43139fc63..ad067e1fdcd8ba85ed8a022901e87bef92fc090b 100644 --- a/generation_config.json +++ b/generation_config.json @@ -1,5 +1,4 @@ { - "bos_token_id": 151643, "do_sample": true, "eos_token_id": [ 151645, @@ -7,8 +6,8 @@ ], "pad_token_id": 151643, "temperature": 0.7, - "top_p": 0.95, "top_k": 20, - "repetition_penalty": 1.03, - "transformers_version": "4.51.0" + "top_p": 0.95, + "transformers_version": "5.8.0", + "bos_token_id": 151643 } \ No newline at end of file diff --git a/model.safetensors b/model.safetensors index a9923328aed6324f3b8bc4b96ecf4cc43763ec69..2d01a55a8c76a46ec1d94126ad94dfe8a1bc4c29 100644 --- a/model.safetensors +++ b/model.safetensors @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9179fe220727db304cb809795bd582e6dbc90a763ef3579708273b8b09858e5 -size 3441185608 +oid sha256:23c085561ee4f5825163d83ad9fc5cc7accfaac86a241a96d54426232a45e20e +size 6882335328 diff --git a/split_info.json b/split_info.json new file mode 100644 index 0000000000000000000000000000000000000000..8b1b0075d0866579f1bce80b961a23921c0f0863 --- /dev/null +++ b/split_info.json @@ -0,0 +1,28 @@ +{ + "experiment_name": "exp6_sft_numinamath_dpo", + "config_file": "/scratch/jennifer/standard-project-m2-the-transformers/configs/exp6_dpo.yaml", + "resolved_config": { + "num_train_epochs": 1, + "learning_rate": 5e-07, + "per_device_train_batch_size": 1, + "per_device_eval_batch_size": 1, + "gradient_accumulation_steps": 8, + "warmup_ratio": 0.1, + "seed": 42, + "save_total_limit": 3, + "eval_steps": 50, + "logging_steps": 10, + "beta": 0.1, + "loss_type": "sigmoid", + "max_length": 4096, + "max_completion_length": 3072, + "val_fraction": 0.1, + "wandb_project": "cs552-math-dpo", + "experiment_name": "exp6_sft_numinamath_dpo", + "output_dir": "/scratch/checkpoints/exp6_sft_numinamath_dpo", + "sft_checkpoint": "/scratch/checkpoints/exp6_sft_numinamath", + "train_source": "/scratch/data/dpo_pairs_v2/pairs.jsonl" + }, + "n_train": 5048, + "n_val": 561 +} \ No newline at end of file diff --git a/tokenizer_config.json b/tokenizer_config.json index 480b16d724b41320e1e217f2408c04a8a8e326fc..c5b200ff35c4189eb88a66e3570d7477243f6739 100644 --- a/tokenizer_config.json +++ b/tokenizer_config.json @@ -26,6 +26,5 @@ "pad_token": "<|endoftext|>", "split_special_tokens": false, "tokenizer_class": "Qwen2Tokenizer", - "unk_token": null, - "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {#- BOXING INSTRUCTION injected so it is present even when the caller (e.g. CI)\n passes no system message. Targets our no-box / multi-box failure mode. #}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- 'When solving a math problem, work through it step by step and write your final answer inside a \\boxed{...}.<|im_end|>\\n' }}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if loop.index0 > ns.last_query_index %}\n {%- if loop.last or (not loop.last and reasoning_content) %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}" + "unk_token": null } \ No newline at end of file diff --git a/trainer_state.json b/trainer_state.json new file mode 100644 index 0000000000000000000000000000000000000000..9bd4137a09212f2b1a062e2ff126a39d8970b8cb --- /dev/null +++ b/trainer_state.json @@ -0,0 +1,6389 @@ +{ + "best_global_step": 750, + "best_metric": 1.6413334274291993, + "best_model_checkpoint": "/scratch/checkpoints/exp8_grpo_fast/checkpoint-750", + "epoch": 0.273224043715847, + "eval_steps": 150, + "global_step": 750, + "is_hyper_param_search": false, + "is_local_process_zero": true, + "is_world_process_zero": true, + "log_history": [ + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333432674407, + "completions/max_length": 1388.2, + "completions/max_terminated_length": 1149.4, + "completions/mean_length": 961.8666870117188, + "completions/mean_terminated_length": 813.0800170898438, + "completions/min_length": 593.6, + "completions/min_terminated_length": 593.6, + "entropy": 0.15543196325500805, + "epoch": 0.0018214936247723133, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.04320038482546806, + "kl": 0.0001638038011151366, + "learning_rate": 1.45985401459854e-08, + "loss": -0.002662058547139168, + "num_tokens": 30920.0, + "reward": 2.02333345413208, + "reward_std": 0.5894708633422852, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.46666666567325593, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.5233333643525839, + "rewards/multi_component_reward/std": 0.4098228693008423, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12532416619360448, + "sampling/importance_sampling_ratio/mean": 0.04354772977530956, + "sampling/importance_sampling_ratio/min": 0.006865633289925732, + "sampling/sampling_logp_difference/max": 0.305552613735199, + "sampling/sampling_logp_difference/mean": 0.010744557529687882, + "step": 5, + "step_time": 32.99731477890164 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1657.0, + "completions/max_terminated_length": 1145.8, + "completions/mean_length": 919.3000366210938, + "completions/mean_terminated_length": 807.0200317382812, + "completions/min_length": 442.0, + "completions/min_terminated_length": 442.0, + "entropy": 0.21022272085150082, + "epoch": 0.0036429872495446266, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.014283710159361362, + "kl": 0.00032655518104244646, + "learning_rate": 3.2846715328467156e-08, + "loss": -0.0015700791031122209, + "num_tokens": 61163.0, + "reward": 1.180000114440918, + "reward_std": 0.4240097999572754, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.06666666865348816, + "rewards/correctness/std": 0.1632993221282959, + "rewards/multi_component_reward/mean": 0.11333334147930145, + "rewards/multi_component_reward/std": 0.27013830840587616, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.06362589932978154, + "sampling/importance_sampling_ratio/mean": 0.01664090696722269, + "sampling/importance_sampling_ratio/min": 0.0005761486502072345, + "sampling/sampling_logp_difference/max": 0.49286861419677735, + "sampling/sampling_logp_difference/mean": 0.01417195163667202, + "step": 10, + "step_time": 37.58825172036886 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1657.4, + "completions/max_terminated_length": 899.0, + "completions/mean_length": 897.2000244140625, + "completions/mean_terminated_length": 615.4466735839844, + "completions/min_length": 296.4, + "completions/min_terminated_length": 296.4, + "entropy": 0.20501985649267832, + "epoch": 0.00546448087431694, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.0192733071744442, + "kl": 0.0003123952769480335, + "learning_rate": 5.10948905109489e-08, + "loss": -0.004246947914361953, + "num_tokens": 93923.0, + "reward": 1.8200001120567322, + "reward_std": 0.5815625011920929, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.4000000059604645, + "rewards/correctness/std": 0.20655910968780516, + "rewards/multi_component_reward/mean": 0.42000001221895217, + "rewards/multi_component_reward/std": 0.3785122692584991, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.18622441291809083, + "sampling/importance_sampling_ratio/mean": 0.04659921806305647, + "sampling/importance_sampling_ratio/min": 0.0010122977004923522, + "sampling/sampling_logp_difference/max": 0.39253207445144656, + "sampling/sampling_logp_difference/mean": 0.01371441949158907, + "step": 15, + "step_time": 38.31680890209973 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 2023.8, + "completions/max_terminated_length": 1348.6, + "completions/mean_length": 1195.0000610351562, + "completions/mean_terminated_length": 1013.5533630371094, + "completions/min_length": 736.4, + "completions/min_terminated_length": 736.4, + "entropy": 0.15656836417814096, + "epoch": 0.007285974499089253, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.02143455669283867, + "kl": 0.00023284607256452244, + "learning_rate": 6.934306569343065e-08, + "loss": -0.006335017085075378, + "num_tokens": 131981.0, + "reward": 1.3033334732055664, + "reward_std": 0.7852452993392944, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.3265986442565918, + "rewards/multi_component_reward/mean": 0.1366666793823242, + "rewards/multi_component_reward/std": 0.5012870132923126, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.05830998048186302, + "sampling/importance_sampling_ratio/mean": 0.016274437960237264, + "sampling/importance_sampling_ratio/min": 0.00017535050738298644, + "sampling/sampling_logp_difference/max": 0.3216116189956665, + "sampling/sampling_logp_difference/mean": 0.011110224667936564, + "step": 20, + "step_time": 45.75825558342039 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1375.4, + "completions/max_terminated_length": 1177.4, + "completions/mean_length": 1042.1333740234375, + "completions/mean_terminated_length": 911.9000244140625, + "completions/min_length": 680.6, + "completions/min_terminated_length": 680.6, + "entropy": 0.1741524614393711, + "epoch": 0.009107468123861567, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.01335141435265541, + "kl": 0.0002629145164974034, + "learning_rate": 8.759124087591241e-08, + "loss": 0.0026412704959511758, + "num_tokens": 166221.0, + "reward": 1.6033333897590638, + "reward_std": 0.5469928443431854, + "rewards/boxed_rate/mean": 0.8666666746139526, + "rewards/boxed_rate/std": 0.20655910968780516, + "rewards/correctness/mean": 0.26666667461395266, + "rewards/correctness/std": 0.20655910968780516, + "rewards/multi_component_reward/mean": 0.3033333241939545, + "rewards/multi_component_reward/std": 0.36893237233161924, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.20655910968780516, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.04747706279158592, + "sampling/importance_sampling_ratio/mean": 0.016832177247852086, + "sampling/importance_sampling_ratio/min": 0.00119685190729814, + "sampling/sampling_logp_difference/max": 0.3412928104400635, + "sampling/sampling_logp_difference/mean": 0.011194901075214148, + "step": 25, + "step_time": 33.345185896754266 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2666666716337204, + "completions/max_length": 1851.2, + "completions/max_terminated_length": 1397.6, + "completions/mean_length": 1248.2000366210937, + "completions/mean_terminated_length": 1000.1933471679688, + "completions/min_length": 643.4, + "completions/min_terminated_length": 643.4, + "entropy": 0.1906235640247663, + "epoch": 0.01092896174863388, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.02501230686903, + "kl": 0.0002681747629443028, + "learning_rate": 1.0583941605839415e-07, + "loss": -0.0029103375971317293, + "num_tokens": 207351.0, + "reward": 1.3366667747497558, + "reward_std": 0.7248244881629944, + "rewards/boxed_rate/mean": 0.7333333313465118, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.10333333685994148, + "rewards/multi_component_reward/std": 0.48569379448890687, + "rewards/no_answer_rate/mean": 0.2666666716337204, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.05718064345419407, + "sampling/importance_sampling_ratio/mean": 0.01394880290608853, + "sampling/importance_sampling_ratio/min": 3.71510857371279e-06, + "sampling/sampling_logp_difference/max": 0.39807581901550293, + "sampling/sampling_logp_difference/mean": 0.011931476183235645, + "step": 30, + "step_time": 42.76963838078082 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1265.8, + "completions/max_terminated_length": 1010.8, + "completions/mean_length": 801.2000183105469, + "completions/mean_terminated_length": 710.2333435058594, + "completions/min_length": 515.6, + "completions/min_terminated_length": 515.6, + "entropy": 0.18569878712296486, + "epoch": 0.012750455373406194, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.013473724946379662, + "kl": 0.00025912571412239536, + "learning_rate": 1.240875912408759e-07, + "loss": -0.007362464070320129, + "num_tokens": 234243.0, + "reward": 1.5500000715255737, + "reward_std": 0.6336740732192994, + "rewards/boxed_rate/mean": 0.9333333373069763, + "rewards/boxed_rate/std": 0.10327955484390258, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.3166666716337204, + "rewards/multi_component_reward/std": 0.35141916275024415, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.10327955484390258, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.10159578723832965, + "sampling/importance_sampling_ratio/mean": 0.03656362611800432, + "sampling/importance_sampling_ratio/min": 0.002998881617435645, + "sampling/sampling_logp_difference/max": 0.3329684495925903, + "sampling/sampling_logp_difference/mean": 0.011683504190295934, + "step": 35, + "step_time": 29.749523213878273 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333334028720856, + "completions/max_length": 1788.4, + "completions/max_terminated_length": 1253.4, + "completions/mean_length": 1151.96669921875, + "completions/mean_terminated_length": 852.2566772460938, + "completions/min_length": 632.2, + "completions/min_terminated_length": 632.2, + "entropy": 0.19733075598875682, + "epoch": 0.014571948998178506, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.017843080684542656, + "kl": 0.00030036380388385927, + "learning_rate": 1.4233576642335764e-07, + "loss": 0.0052158843725919725, + "num_tokens": 271334.0, + "reward": 1.3800000905990601, + "reward_std": 0.8547543048858642, + "rewards/boxed_rate/mean": 0.7666666686534882, + "rewards/boxed_rate/std": 0.2128240704536438, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.35449349880218506, + "rewards/multi_component_reward/mean": 0.1800000086426735, + "rewards/multi_component_reward/std": 0.5035018444061279, + "rewards/no_answer_rate/mean": 0.2333333373069763, + "rewards/no_answer_rate/std": 0.2128240704536438, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.05587816089391708, + "sampling/importance_sampling_ratio/mean": 0.020892591564916074, + "sampling/importance_sampling_ratio/min": 2.3939082747448788e-05, + "sampling/sampling_logp_difference/max": 0.42613508701324465, + "sampling/sampling_logp_difference/mean": 0.012935483455657959, + "step": 40, + "step_time": 40.51654889807105 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1250.0, + "completions/max_terminated_length": 700.2, + "completions/mean_length": 596.7666748046875, + "completions/mean_terminated_length": 492.7933410644531, + "completions/min_length": 341.8, + "completions/min_terminated_length": 341.8, + "entropy": 0.22809130648771922, + "epoch": 0.01639344262295082, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.06362088024616241, + "kl": 0.00031622202028908455, + "learning_rate": 1.6058394160583942e-07, + "loss": -0.0004649309907108545, + "num_tokens": 293263.0, + "reward": 1.6900001049041748, + "reward_std": 0.6551559090614318, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.3000000089406967, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.3900000035762787, + "rewards/multi_component_reward/std": 0.37366979122161864, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.06921428106725216, + "sampling/importance_sampling_ratio/mean": 0.029317377135157585, + "sampling/importance_sampling_ratio/min": 0.00602934339824146, + "sampling/sampling_logp_difference/max": 0.35652687549591067, + "sampling/sampling_logp_difference/mean": 0.015356982313096523, + "step": 45, + "step_time": 28.76009795963764 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1886.2, + "completions/max_terminated_length": 1599.0, + "completions/mean_length": 1163.9333862304688, + "completions/mean_terminated_length": 1057.6466918945312, + "completions/min_length": 663.8, + "completions/min_terminated_length": 663.8, + "entropy": 0.17236283471186956, + "epoch": 0.018214936247723135, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01997266709804535, + "kl": 0.0002550489671799975, + "learning_rate": 1.7883211678832117e-07, + "loss": -0.005824955180287361, + "num_tokens": 331739.0, + "reward": 1.8566666960716247, + "reward_std": 0.7818804442882538, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.36666667461395264, + "rewards/correctness/std": 0.3161036252975464, + "rewards/multi_component_reward/mean": 0.4233333319425583, + "rewards/multi_component_reward/std": 0.4823453426361084, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.054672015644609925, + "sampling/importance_sampling_ratio/mean": 0.01720548253506422, + "sampling/importance_sampling_ratio/min": 0.00035658250299805546, + "sampling/sampling_logp_difference/max": 0.3509830951690674, + "sampling/sampling_logp_difference/mean": 0.011117835808545352, + "step": 50, + "step_time": 43.21503445059061 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.3333333432674408, + "completions/max_length": 1744.8, + "completions/max_terminated_length": 1236.2, + "completions/mean_length": 1297.1666931152345, + "completions/mean_terminated_length": 1065.0066833496094, + "completions/min_length": 936.6, + "completions/min_terminated_length": 936.6, + "entropy": 0.12224519066512585, + "epoch": 0.020036429872495445, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.021082526072859764, + "kl": 0.00019305126891898302, + "learning_rate": 1.9708029197080292e-07, + "loss": 0.0025430308654904366, + "num_tokens": 373144.0, + "reward": 1.4600000858306885, + "reward_std": 0.7388215482234954, + "rewards/boxed_rate/mean": 0.6666666597127915, + "rewards/boxed_rate/std": 0.35449349880218506, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.12666668593883515, + "rewards/multi_component_reward/std": 0.506528252363205, + "rewards/no_answer_rate/mean": 0.33333333134651183, + "rewards/no_answer_rate/std": 0.35449349880218506, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.18492921590805053, + "sampling/importance_sampling_ratio/max": 0.10442680462729186, + "sampling/importance_sampling_ratio/mean": 0.04177726461493876, + "sampling/importance_sampling_ratio/min": 0.011532440476071316, + "sampling/sampling_logp_difference/max": 0.32466731071472166, + "sampling/sampling_logp_difference/mean": 0.007843528036028146, + "step": 55, + "step_time": 40.733187234401704 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333432674407, + "completions/max_length": 1700.0, + "completions/max_terminated_length": 1413.4, + "completions/mean_length": 923.9000366210937, + "completions/mean_terminated_length": 779.9666870117187, + "completions/min_length": 489.0, + "completions/min_terminated_length": 489.0, + "entropy": 0.18838655066986879, + "epoch": 0.02185792349726776, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.016334548592567444, + "kl": 0.0002971158295016115, + "learning_rate": 2.1532846715328465e-07, + "loss": 0.0031619951128959655, + "num_tokens": 403573.0, + "reward": 1.393333411216736, + "reward_std": 0.45263335704803465, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.19333333075046538, + "rewards/multi_component_reward/std": 0.30624626874923705, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.14395143389701842, + "sampling/importance_sampling_ratio/mean": 0.04990396222565323, + "sampling/importance_sampling_ratio/min": 0.00032090346864226403, + "sampling/sampling_logp_difference/max": 0.40355303287506106, + "sampling/sampling_logp_difference/mean": 0.012135114334523679, + "step": 60, + "step_time": 38.38553868718445 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1554.4, + "completions/max_terminated_length": 1147.6, + "completions/mean_length": 972.8000244140625, + "completions/mean_terminated_length": 804.5466735839843, + "completions/min_length": 498.2, + "completions/min_terminated_length": 498.2, + "entropy": 0.1604183206955592, + "epoch": 0.023679417122040074, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.00623254245147109, + "kl": 0.00027490937888311845, + "learning_rate": 2.335766423357664e-07, + "loss": -0.005589158460497856, + "num_tokens": 436225.0, + "reward": 1.490000057220459, + "reward_std": 0.6202400684356689, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.19000000208616258, + "rewards/multi_component_reward/std": 0.4335045665502548, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.1, + "rewards/repetition_rate/std": 0.10954451560974121, + "sampling/importance_sampling_ratio/max": 0.0990872667171061, + "sampling/importance_sampling_ratio/mean": 0.04436926119960845, + "sampling/importance_sampling_ratio/min": 0.004196830893285148, + "sampling/sampling_logp_difference/max": 0.297141432762146, + "sampling/sampling_logp_difference/mean": 0.010508796572685242, + "step": 65, + "step_time": 36.07466428950429 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1634.0, + "completions/max_terminated_length": 994.8, + "completions/mean_length": 744.2666870117188, + "completions/mean_terminated_length": 601.7000122070312, + "completions/min_length": 374.2, + "completions/min_terminated_length": 374.2, + "entropy": 0.2318819284439087, + "epoch": 0.025500910746812388, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05441533029079437, + "kl": 0.0003520799073157832, + "learning_rate": 2.518248175182482e-07, + "loss": -0.003310435265302658, + "num_tokens": 460845.0, + "reward": 1.6933334827423097, + "reward_std": 0.7755845502018929, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.3, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.36000003293156624, + "rewards/multi_component_reward/std": 0.5008985161781311, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.056702688336372375, + "sampling/importance_sampling_ratio/mean": 0.019475685060024263, + "sampling/importance_sampling_ratio/min": 0.001143362923176032, + "sampling/sampling_logp_difference/max": 0.3391279220581055, + "sampling/sampling_logp_difference/mean": 0.014528140239417554, + "step": 70, + "step_time": 36.275424292869864 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1515.2, + "completions/max_terminated_length": 845.0, + "completions/mean_length": 917.0000244140625, + "completions/mean_terminated_length": 678.5633483886719, + "completions/min_length": 475.4, + "completions/min_terminated_length": 475.4, + "entropy": 0.16173700019717216, + "epoch": 0.0273224043715847, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018244028091430664, + "kl": 0.00024622106284368785, + "learning_rate": 2.700729927007299e-07, + "loss": -0.014353486895561218, + "num_tokens": 491205.0, + "reward": 1.633333432674408, + "reward_std": 0.8448223888874054, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.3, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.3000000298023224, + "rewards/multi_component_reward/std": 0.5207933127880097, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.16285581663250923, + "sampling/importance_sampling_ratio/mean": 0.05478177797049284, + "sampling/importance_sampling_ratio/min": 0.0013068500558716862, + "sampling/sampling_logp_difference/max": 0.3721138000488281, + "sampling/sampling_logp_difference/mean": 0.011166188679635525, + "step": 75, + "step_time": 35.12463851571083 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1582.6, + "completions/max_terminated_length": 1313.0, + "completions/mean_length": 985.9666870117187, + "completions/mean_terminated_length": 935.2533569335938, + "completions/min_length": 670.8, + "completions/min_terminated_length": 670.8, + "entropy": 0.17678433768451213, + "epoch": 0.029143897996357013, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.009146427735686302, + "kl": 0.0002533862963900901, + "learning_rate": 2.8832116788321166e-07, + "loss": 0.002100883610546589, + "num_tokens": 524180.0, + "reward": 1.6633334636688233, + "reward_std": 0.8858610212802887, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.3823883533477783, + "rewards/multi_component_reward/mean": 0.36333334222435953, + "rewards/multi_component_reward/std": 0.473619781434536, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.044011454284191134, + "sampling/importance_sampling_ratio/mean": 0.013894069381058216, + "sampling/importance_sampling_ratio/min": 0.001124254113574473, + "sampling/sampling_logp_difference/max": 0.32026147842407227, + "sampling/sampling_logp_difference/mean": 0.01106615299358964, + "step": 80, + "step_time": 36.22448882814497 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1693.6, + "completions/max_terminated_length": 1402.8, + "completions/mean_length": 1091.4666870117187, + "completions/mean_terminated_length": 998.7600219726562, + "completions/min_length": 584.2, + "completions/min_terminated_length": 584.2, + "entropy": 0.1462914695342382, + "epoch": 0.030965391621129327, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018090059980750084, + "kl": 0.00022486963668294873, + "learning_rate": 3.065693430656934e-07, + "loss": -0.00974438264966011, + "num_tokens": 560254.0, + "reward": 1.8100001335144043, + "reward_std": 1.0823543906211852, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.36666667461395264, + "rewards/correctness/std": 0.4794029474258423, + "rewards/multi_component_reward/mean": 0.44333334267139435, + "rewards/multi_component_reward/std": 0.6124810755252839, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.1276155561208725, + "sampling/importance_sampling_ratio/mean": 0.03584536015987396, + "sampling/importance_sampling_ratio/min": 0.002319600686418858, + "sampling/sampling_logp_difference/max": 0.41160542964935304, + "sampling/sampling_logp_difference/mean": 0.010097185987979174, + "step": 85, + "step_time": 39.475563449971375 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.3333333432674408, + "completions/max_length": 1841.2, + "completions/max_terminated_length": 875.0, + "completions/mean_length": 1059.2000122070312, + "completions/mean_terminated_length": 630.0533386230469, + "completions/min_length": 458.8, + "completions/min_terminated_length": 458.8, + "entropy": 0.2378736046453317, + "epoch": 0.03278688524590164, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.00910874642431736, + "kl": 0.0003503820747331095, + "learning_rate": 3.2481751824817516e-07, + "loss": 0.0034383241087198257, + "num_tokens": 593668.0, + "reward": 1.4166667222976685, + "reward_std": 0.8338384032249451, + "rewards/boxed_rate/mean": 0.6333333253860474, + "rewards/boxed_rate/std": 0.451508092880249, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.11666668206453323, + "rewards/multi_component_reward/std": 0.566602936387062, + "rewards/no_answer_rate/mean": 0.36666667759418486, + "rewards/no_answer_rate/std": 0.451508092880249, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.05201358497142792, + "sampling/importance_sampling_ratio/mean": 0.017090958543121815, + "sampling/importance_sampling_ratio/min": 0.0002041866974779629, + "sampling/sampling_logp_difference/max": 0.3399634838104248, + "sampling/sampling_logp_difference/mean": 0.015104132890701293, + "step": 90, + "step_time": 41.00809515919536 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1441.6, + "completions/max_terminated_length": 960.8, + "completions/mean_length": 802.6000366210938, + "completions/mean_terminated_length": 711.3800231933594, + "completions/min_length": 525.6, + "completions/min_terminated_length": 525.6, + "entropy": 0.2225553606947263, + "epoch": 0.03460837887067395, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.01798272132873535, + "kl": 0.00032260040970868433, + "learning_rate": 3.4306569343065697e-07, + "loss": 0.007113450020551681, + "num_tokens": 621316.0, + "reward": 1.223333477973938, + "reward_std": 0.36713925525546076, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.06666666865348816, + "rewards/correctness/std": 0.1632993221282959, + "rewards/multi_component_reward/mean": 0.12333333715796471, + "rewards/multi_component_reward/std": 0.2738735854625702, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.15052475407719612, + "sampling/importance_sampling_ratio/mean": 0.046577111538499597, + "sampling/importance_sampling_ratio/min": 0.012816431716431823, + "sampling/sampling_logp_difference/max": 0.34408046007156373, + "sampling/sampling_logp_difference/mean": 0.014901423826813698, + "step": 95, + "step_time": 32.91087112892419 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1537.8, + "completions/max_terminated_length": 1380.2, + "completions/mean_length": 972.9000366210937, + "completions/mean_terminated_length": 902.8400146484375, + "completions/min_length": 618.0, + "completions/min_terminated_length": 618.0, + "entropy": 0.2104227361579736, + "epoch": 0.03642987249544627, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.006778502371162176, + "kl": 0.00031352789907638606, + "learning_rate": 3.6131386861313867e-07, + "loss": -0.0013602237217128278, + "num_tokens": 653443.0, + "reward": 1.880000078678131, + "reward_std": 0.9815195143222809, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.4000000059604645, + "rewards/correctness/std": 0.4256481409072876, + "rewards/multi_component_reward/mean": 0.48000000715255736, + "rewards/multi_component_reward/std": 0.5586783826351166, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.0448111507575959, + "sampling/importance_sampling_ratio/mean": 0.01701541549991816, + "sampling/importance_sampling_ratio/min": 0.0003750977984182488, + "sampling/sampling_logp_difference/max": 0.3470792770385742, + "sampling/sampling_logp_difference/mean": 0.013565080799162388, + "step": 100, + "step_time": 35.31437961217016 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1421.4, + "completions/max_terminated_length": 1072.0, + "completions/mean_length": 947.7000366210938, + "completions/mean_terminated_length": 787.706689453125, + "completions/min_length": 551.6, + "completions/min_terminated_length": 551.6, + "entropy": 0.2239283885806799, + "epoch": 0.03825136612021858, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.011714774183928967, + "kl": 0.0003353157648234628, + "learning_rate": 3.795620437956204e-07, + "loss": 0.009437119215726852, + "num_tokens": 683674.0, + "reward": 1.4433334589004516, + "reward_std": 0.8665581226348877, + "rewards/boxed_rate/mean": 0.8333333313465119, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.21000000834465027, + "rewards/multi_component_reward/std": 0.5039317846298218, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.1078285550349392, + "sampling/importance_sampling_ratio/mean": 0.03690805228543468, + "sampling/importance_sampling_ratio/min": 0.0026856323238462275, + "sampling/sampling_logp_difference/max": 0.41998746395111086, + "sampling/sampling_logp_difference/mean": 0.014628523960709572, + "step": 105, + "step_time": 32.93989271316677 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1471.4, + "completions/max_terminated_length": 979.4, + "completions/mean_length": 852.2666748046875, + "completions/mean_terminated_length": 721.6166748046875, + "completions/min_length": 488.4, + "completions/min_terminated_length": 488.4, + "entropy": 0.14421232057114441, + "epoch": 0.04007285974499089, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.07266412675380707, + "kl": 0.00022867745647090488, + "learning_rate": 3.978102189781022e-07, + "loss": 0.0018386229872703551, + "num_tokens": 713286.0, + "reward": 1.6733334064483643, + "reward_std": 0.7425484180450439, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.3161036252975464, + "rewards/multi_component_reward/mean": 0.3400000125169754, + "rewards/multi_component_reward/std": 0.47312636375427247, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.13587955459952356, + "sampling/importance_sampling_ratio/mean": 0.050121305510401726, + "sampling/importance_sampling_ratio/min": 0.0025831204319047172, + "sampling/sampling_logp_difference/max": 0.345587694644928, + "sampling/sampling_logp_difference/mean": 0.00959718506783247, + "step": 110, + "step_time": 33.89908826816827 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2333333373069763, + "completions/max_length": 1749.4, + "completions/max_terminated_length": 884.2, + "completions/mean_length": 869.533349609375, + "completions/mean_terminated_length": 539.5833374023438, + "completions/min_length": 307.8, + "completions/min_terminated_length": 307.8, + "entropy": 0.2536199669043223, + "epoch": 0.04189435336976321, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.011132585816085339, + "kl": 0.0003656693840942656, + "learning_rate": 4.160583941605839e-07, + "loss": 0.0021231153979897497, + "num_tokens": 741952.0, + "reward": 1.7500001192092896, + "reward_std": 1.106396782398224, + "rewards/boxed_rate/mean": 0.799999988079071, + "rewards/boxed_rate/std": 0.35449349880218506, + "rewards/correctness/mean": 0.36666667461395264, + "rewards/correctness/std": 0.4794029474258423, + "rewards/multi_component_reward/mean": 0.383333346247673, + "rewards/multi_component_reward/std": 0.6433047533035279, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.35449349880218506, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.25683521628379824, + "sampling/importance_sampling_ratio/mean": 0.065315605327487, + "sampling/importance_sampling_ratio/min": 0.00023330498968689221, + "sampling/sampling_logp_difference/max": 0.35823286771774293, + "sampling/sampling_logp_difference/mean": 0.01769500244408846, + "step": 115, + "step_time": 39.532602636888626 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1573.0, + "completions/max_terminated_length": 1324.8, + "completions/mean_length": 1093.2000366210937, + "completions/mean_terminated_length": 1037.0133666992188, + "completions/min_length": 766.8, + "completions/min_terminated_length": 766.8, + "entropy": 0.15541148359576862, + "epoch": 0.04371584699453552, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.01879267208278179, + "kl": 0.00025834220529456314, + "learning_rate": 4.343065693430657e-07, + "loss": 0.0006856221705675125, + "num_tokens": 777766.0, + "reward": 1.433333396911621, + "reward_std": 0.4934782743453979, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2128240704536438, + "rewards/multi_component_reward/mean": 0.23333334028720856, + "rewards/multi_component_reward/std": 0.32306827008724215, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.027404608484357596, + "sampling/importance_sampling_ratio/mean": 0.012392069399356841, + "sampling/importance_sampling_ratio/min": 0.0004944111146869545, + "sampling/sampling_logp_difference/max": 0.37873687744140627, + "sampling/sampling_logp_difference/mean": 0.010291843488812447, + "step": 120, + "step_time": 36.481429217197004 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1672.2, + "completions/max_terminated_length": 1231.2, + "completions/mean_length": 1009.0000122070312, + "completions/mean_terminated_length": 875.0133422851562, + "completions/min_length": 573.0, + "completions/min_terminated_length": 573.0, + "entropy": 0.16983376170198122, + "epoch": 0.04553734061930783, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.026680519804358482, + "kl": 0.00028378382169951997, + "learning_rate": 4.5255474452554743e-07, + "loss": 0.002333539165556431, + "num_tokens": 811096.0, + "reward": 1.3900000691413879, + "reward_std": 0.4993088662624359, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.22333332896232605, + "rewards/multi_component_reward/std": 0.31437968015670775, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.1588123269379139, + "sampling/importance_sampling_ratio/mean": 0.04118972420692444, + "sampling/importance_sampling_ratio/min": 0.002068098068754896, + "sampling/sampling_logp_difference/max": 0.42397408485412597, + "sampling/sampling_logp_difference/mean": 0.01178776090964675, + "step": 125, + "step_time": 38.221346308663485 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1774.8, + "completions/max_terminated_length": 1017.6, + "completions/mean_length": 970.7666870117188, + "completions/mean_terminated_length": 722.2733520507812, + "completions/min_length": 472.0, + "completions/min_terminated_length": 472.0, + "entropy": 0.20842353130380312, + "epoch": 0.04735883424408015, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.017246264964342117, + "kl": 0.00032587918249191714, + "learning_rate": 4.7080291970802913e-07, + "loss": -0.004489587619900703, + "num_tokens": 843681.0, + "reward": 1.3233334302902222, + "reward_std": 0.6886623933911323, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.12333333268761634, + "rewards/multi_component_reward/std": 0.4415378957986832, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.057003194093704225, + "sampling/importance_sampling_ratio/mean": 0.015649724379181863, + "sampling/importance_sampling_ratio/min": 0.0008366983823620787, + "sampling/sampling_logp_difference/max": 0.42825262546539306, + "sampling/sampling_logp_difference/mean": 0.013275405205786229, + "step": 130, + "step_time": 40.03070425353944 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1638.0, + "completions/max_terminated_length": 1082.6, + "completions/mean_length": 965.3000427246094, + "completions/mean_terminated_length": 805.8833435058593, + "completions/min_length": 475.4, + "completions/min_terminated_length": 475.4, + "entropy": 0.20910785247882208, + "epoch": 0.04918032786885246, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.014268649742007256, + "kl": 0.00030581061485766744, + "learning_rate": 4.89051094890511e-07, + "loss": -0.0031251441687345505, + "num_tokens": 875316.0, + "reward": 1.3700001120567322, + "reward_std": 0.6670311987400055, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.20333334803581238, + "rewards/multi_component_reward/std": 0.4036152184009552, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.051390893710777166, + "sampling/importance_sampling_ratio/mean": 0.015738325094571338, + "sampling/importance_sampling_ratio/min": 0.0017150626291213885, + "sampling/sampling_logp_difference/max": 0.3242307841777802, + "sampling/sampling_logp_difference/mean": 0.01380961835384369, + "step": 135, + "step_time": 37.537090591713785 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333334028720856, + "completions/max_length": 1763.0, + "completions/max_terminated_length": 775.6, + "completions/mean_length": 960.7000122070312, + "completions/mean_terminated_length": 604.2866760253906, + "completions/min_length": 455.4, + "completions/min_terminated_length": 455.4, + "entropy": 0.23928763965765634, + "epoch": 0.051001821493624776, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.037585072219371796, + "kl": 0.0004156571065929408, + "learning_rate": 4.996165644171779e-07, + "loss": -0.003277498111128807, + "num_tokens": 907599.0, + "reward": 1.8700001239776611, + "reward_std": 1.213301706314087, + "rewards/boxed_rate/mean": 0.7666666567325592, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.4333333343267441, + "rewards/correctness/std": 0.5198277235031128, + "rewards/multi_component_reward/mean": 0.4366666838526726, + "rewards/multi_component_reward/std": 0.7051358222961426, + "rewards/no_answer_rate/mean": 0.23333334028720856, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.03320965538732708, + "sampling/importance_sampling_ratio/mean": 0.015124964842107147, + "sampling/importance_sampling_ratio/min": 0.0001439221122021143, + "sampling/sampling_logp_difference/max": 0.3350650787353516, + "sampling/sampling_logp_difference/mean": 0.015135842747986317, + "step": 140, + "step_time": 39.69141132887453 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333432674407, + "completions/max_length": 1378.6, + "completions/max_terminated_length": 1035.8, + "completions/mean_length": 898.6333557128906, + "completions/mean_terminated_length": 752.8266784667969, + "completions/min_length": 510.2, + "completions/min_terminated_length": 510.2, + "entropy": 0.15787406551341215, + "epoch": 0.052823315118397086, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.19890737533569336, + "kl": 0.0003336386442242656, + "learning_rate": 4.986579754601227e-07, + "loss": 0.002312604896724224, + "num_tokens": 936976.0, + "reward": 1.6733333706855773, + "reward_std": 0.7778201699256897, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.3161036252975464, + "rewards/multi_component_reward/mean": 0.34000000953674314, + "rewards/multi_component_reward/std": 0.4764533966779709, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.17277967557311058, + "sampling/importance_sampling_ratio/mean": 0.0497883933596313, + "sampling/importance_sampling_ratio/min": 0.0009090198537052175, + "sampling/sampling_logp_difference/max": 0.5492818593978882, + "sampling/sampling_logp_difference/mean": 0.010555424820631742, + "step": 145, + "step_time": 32.12684296686202 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1774.2, + "completions/max_terminated_length": 1617.8, + "completions/mean_length": 1212.6333618164062, + "completions/mean_terminated_length": 1111.5633544921875, + "completions/min_length": 845.2, + "completions/min_terminated_length": 845.2, + "entropy": 0.17395392432808876, + "epoch": 0.0546448087431694, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.021323509514331818, + "kl": 0.00030080197708836444, + "learning_rate": 4.976993865030675e-07, + "loss": -0.005566118285059929, + "num_tokens": 976577.0, + "reward": 1.4133334517478944, + "reward_std": 0.5511920034885407, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2128240704536438, + "rewards/multi_component_reward/mean": 0.17999999672174455, + "rewards/multi_component_reward/std": 0.3554683744907379, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.05498660355806351, + "sampling/importance_sampling_ratio/mean": 0.01640337195713073, + "sampling/importance_sampling_ratio/min": 1.4973960100639972e-05, + "sampling/sampling_logp_difference/max": 0.47727389335632325, + "sampling/sampling_logp_difference/mean": 0.011420094780623913, + "step": 150, + "step_time": 41.61420878618956 + }, + { + "epoch": 0.0546448087431694, + "eval_clip_ratio/high_max": 0.0, + "eval_clip_ratio/high_mean": 0.0, + "eval_clip_ratio/low_mean": 0.0, + "eval_clip_ratio/low_min": 0.0, + "eval_clip_ratio/region_mean": 0.0, + "eval_completions/clipped_ratio": 0.21666667133569717, + "eval_completions/max_length": 1789.4, + "eval_completions/max_terminated_length": 1216.88, + "eval_completions/mean_length": 1121.4300323486327, + "eval_completions/mean_terminated_length": 886.213013305664, + "eval_completions/min_length": 641.78, + "eval_completions/min_terminated_length": 641.78, + "eval_entropy": 0.17698929071426392, + "eval_frac_reward_zero_std": 0.08, + "eval_kl": 0.00028998344379942864, + "eval_loss": -0.0026245773769915104, + "eval_num_tokens": 976577.0, + "eval_reward": 1.5393334233760834, + "eval_reward_std": 0.7298788775503635, + "eval_rewards/boxed_rate/mean": 0.7833333307504654, + "eval_rewards/boxed_rate/std": 0.32846659898757935, + "eval_rewards/correctness/mean": 0.24666666924953462, + "eval_rewards/correctness/std": 0.27019834876060483, + "eval_rewards/multi_component_reward/mean": 0.21933334972709417, + "eval_rewards/multi_component_reward/std": 0.48081229120492935, + "eval_rewards/no_answer_rate/mean": 0.21666667133569717, + "eval_rewards/no_answer_rate/std": 0.32846659898757935, + "eval_rewards/repetition_rate/mean": 0.07333333462476731, + "eval_rewards/repetition_rate/std": 0.13300593733787536, + "eval_runtime": 1782.1981, + "eval_samples_per_second": 0.028, + "eval_sampling/importance_sampling_ratio/max": 0.07466922109248117, + "eval_sampling/importance_sampling_ratio/mean": 0.026714042400126346, + "eval_sampling/importance_sampling_ratio/min": 0.0011027993811156295, + "eval_sampling/sampling_logp_difference/max": 0.4103265881538391, + "eval_sampling/sampling_logp_difference/mean": 0.011447538509964944, + "eval_steps_per_second": 0.005, + "step": 150 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 1207.0, + "completions/max_terminated_length": 1207.0, + "completions/mean_length": 770.1333679199219, + "completions/mean_terminated_length": 770.1333679199219, + "completions/min_length": 500.8, + "completions/min_terminated_length": 500.8, + "entropy": 0.19859646161397299, + "epoch": 0.056466302367941715, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.05886993184685707, + "kl": 0.0003771214178414084, + "learning_rate": 4.967407975460122e-07, + "loss": -0.004334203898906708, + "num_tokens": 1003269.0, + "reward": 1.4733334302902221, + "reward_std": 0.504085260629654, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2128240704536438, + "rewards/multi_component_reward/mean": 0.27333334758877753, + "rewards/multi_component_reward/std": 0.2586013779044151, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.10831067697145044, + "sampling/importance_sampling_ratio/mean": 0.040180114656686784, + "sampling/importance_sampling_ratio/min": 0.0008547768986318261, + "sampling/sampling_logp_difference/max": 0.3474395990371704, + "sampling/sampling_logp_difference/mean": 0.013024460710585117, + "step": 155, + "step_time": 28.411310048028827 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1885.8, + "completions/max_terminated_length": 1037.6, + "completions/mean_length": 888.3666748046875, + "completions/mean_terminated_length": 646.7700012207031, + "completions/min_length": 398.6, + "completions/min_terminated_length": 398.6, + "entropy": 0.21153039361039797, + "epoch": 0.058287795992714025, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.030352359637618065, + "kl": 0.00036930939143834014, + "learning_rate": 4.957822085889571e-07, + "loss": -0.015004897117614746, + "num_tokens": 1032692.0, + "reward": 1.6533334136009217, + "reward_std": 0.7879160098731518, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.3161036252975464, + "rewards/multi_component_reward/mean": 0.3200000114738941, + "rewards/multi_component_reward/std": 0.5426497280597686, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.15256396904587746, + "sampling/importance_sampling_ratio/mean": 0.05275641949847341, + "sampling/importance_sampling_ratio/min": 0.0003334702798868202, + "sampling/sampling_logp_difference/max": 0.3136852741241455, + "sampling/sampling_logp_difference/mean": 0.014769792556762695, + "step": 160, + "step_time": 41.816410617157814 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2666666716337204, + "completions/max_length": 1347.0, + "completions/max_terminated_length": 859.2, + "completions/mean_length": 907.3000244140625, + "completions/mean_terminated_length": 642.22001953125, + "completions/min_length": 436.6, + "completions/min_terminated_length": 436.6, + "entropy": 0.2138912024597327, + "epoch": 0.060109289617486336, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.0335945226252079, + "kl": 0.00036641353217419236, + "learning_rate": 4.948236196319019e-07, + "loss": -0.001109707448631525, + "num_tokens": 1062455.0, + "reward": 1.733333444595337, + "reward_std": 0.6965994656085968, + "rewards/boxed_rate/mean": 0.7333333313465118, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.3666666686534882, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.33333335630595684, + "rewards/multi_component_reward/std": 0.4374739408493042, + "rewards/no_answer_rate/mean": 0.2666666716337204, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.17867551841773094, + "sampling/importance_sampling_ratio/mean": 0.07929584998637437, + "sampling/importance_sampling_ratio/min": 0.007277285696347713, + "sampling/sampling_logp_difference/max": 0.3139556646347046, + "sampling/sampling_logp_difference/mean": 0.014788956940174102, + "step": 165, + "step_time": 31.849167810566723 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1693.4, + "completions/max_terminated_length": 1086.8, + "completions/mean_length": 985.8000244140625, + "completions/mean_terminated_length": 765.3133666992187, + "completions/min_length": 539.0, + "completions/min_terminated_length": 539.0, + "entropy": 0.22184887006878853, + "epoch": 0.061930783242258654, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.010766901075839996, + "kl": 0.0003837254077855808, + "learning_rate": 4.938650306748465e-07, + "loss": -0.005718140304088593, + "num_tokens": 1094465.0, + "reward": 1.5600001335144043, + "reward_std": 0.6616819977760315, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.2666666656732559, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.2933333650231361, + "rewards/multi_component_reward/std": 0.4033259034156799, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.04150749985128641, + "sampling/importance_sampling_ratio/mean": 0.017618318973109125, + "sampling/importance_sampling_ratio/min": 0.0001386483045517349, + "sampling/sampling_logp_difference/max": 0.41498665809631347, + "sampling/sampling_logp_difference/mean": 0.015049760602414608, + "step": 170, + "step_time": 38.327568624168634 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.33333334028720857, + "completions/max_length": 2048.0, + "completions/max_terminated_length": 1280.8, + "completions/mean_length": 1264.76669921875, + "completions/mean_terminated_length": 950.460009765625, + "completions/min_length": 712.6, + "completions/min_terminated_length": 712.6, + "entropy": 0.20655053692559402, + "epoch": 0.06375227686703097, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.1300213485956192, + "kl": 0.00040407002428158496, + "learning_rate": 4.929064417177914e-07, + "loss": -0.002616160549223423, + "num_tokens": 1134790.0, + "reward": 1.2733334183692933, + "reward_std": 0.7226614370942116, + "rewards/boxed_rate/mean": 0.6666666567325592, + "rewards/boxed_rate/std": 0.45777305364608767, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.0733333457261324, + "rewards/multi_component_reward/std": 0.5279393553733825, + "rewards/no_answer_rate/mean": 0.33333334028720857, + "rewards/no_answer_rate/std": 0.45777305364608767, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.05303769689053297, + "sampling/importance_sampling_ratio/mean": 0.015054715669248254, + "sampling/importance_sampling_ratio/min": 8.732383148210488e-05, + "sampling/sampling_logp_difference/max": 0.29345667362213135, + "sampling/sampling_logp_difference/mean": 0.013776615634560585, + "step": 175, + "step_time": 46.28360453415662 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2333333373069763, + "completions/max_length": 1848.8, + "completions/max_terminated_length": 1419.8, + "completions/mean_length": 1218.3667114257812, + "completions/mean_terminated_length": 983.3333618164063, + "completions/min_length": 721.0, + "completions/min_terminated_length": 721.0, + "entropy": 0.14731517334779104, + "epoch": 0.06557377049180328, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.03868393227458, + "kl": 0.00034544099568544574, + "learning_rate": 4.919478527607362e-07, + "loss": -0.007346557080745697, + "num_tokens": 1175133.0, + "reward": 1.7500000834465026, + "reward_std": 0.9921784222126007, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.3666666716337204, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.383333346247673, + "rewards/multi_component_reward/std": 0.5928741514682769, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.08876338973641396, + "sampling/importance_sampling_ratio/mean": 0.02539586406201124, + "sampling/importance_sampling_ratio/min": 0.0011980376816850713, + "sampling/sampling_logp_difference/max": 0.3423057317733765, + "sampling/sampling_logp_difference/mean": 0.010648915357887746, + "step": 180, + "step_time": 42.86893743276596 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 1706.6, + "completions/max_terminated_length": 1151.0, + "completions/mean_length": 1097.7666870117187, + "completions/mean_terminated_length": 885.9666931152344, + "completions/min_length": 678.4, + "completions/min_terminated_length": 678.4, + "entropy": 0.13618133092919985, + "epoch": 0.06739526411657559, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.04725215584039688, + "kl": 0.0003020782351086382, + "learning_rate": 4.909892638036809e-07, + "loss": -0.0009320625104010105, + "num_tokens": 1210850.0, + "reward": 1.5866667985916139, + "reward_std": 0.3331963121891022, + "rewards/boxed_rate/mean": 0.7999999940395355, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.10327955484390258, + "rewards/multi_component_reward/mean": 0.2533333495259285, + "rewards/multi_component_reward/std": 0.27605399787425994, + "rewards/no_answer_rate/mean": 0.20000000596046447, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.0795806871727109, + "sampling/importance_sampling_ratio/mean": 0.02610416170209646, + "sampling/importance_sampling_ratio/min": 0.002741426149717241, + "sampling/sampling_logp_difference/max": 0.3459432005882263, + "sampling/sampling_logp_difference/mean": 0.008953130897134542, + "step": 185, + "step_time": 39.274089214019476 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1897.2, + "completions/max_terminated_length": 1270.2, + "completions/mean_length": 1048.533349609375, + "completions/mean_terminated_length": 838.5800048828125, + "completions/min_length": 490.8, + "completions/min_terminated_length": 490.8, + "entropy": 0.18140921418865522, + "epoch": 0.0692167577413479, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05130758509039879, + "kl": 0.00038463542005047203, + "learning_rate": 4.900306748466257e-07, + "loss": -0.00950215756893158, + "num_tokens": 1245174.0, + "reward": 1.2133333802223205, + "reward_std": 0.6049932837486267, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.36985843181610106, + "rewards/correctness/mean": 0.10000000298023223, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.08000001087784767, + "rewards/multi_component_reward/std": 0.39686688780784607, + "rewards/no_answer_rate/mean": 0.20000000596046447, + "rewards/no_answer_rate/std": 0.36985843181610106, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.0888688538223505, + "sampling/importance_sampling_ratio/mean": 0.03125911168754101, + "sampling/importance_sampling_ratio/min": 0.00010974614788835727, + "sampling/sampling_logp_difference/max": 0.3610305070877075, + "sampling/sampling_logp_difference/mean": 0.01223631165921688, + "step": 190, + "step_time": 42.93450187277049 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2333333373069763, + "completions/max_length": 1931.4, + "completions/max_terminated_length": 1245.0, + "completions/mean_length": 1162.5333740234375, + "completions/mean_terminated_length": 899.483349609375, + "completions/min_length": 618.2, + "completions/min_terminated_length": 618.2, + "entropy": 0.1432020000492533, + "epoch": 0.07103825136612021, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.06832960247993469, + "kl": 0.00033801408014066204, + "learning_rate": 4.890720858895706e-07, + "loss": -0.00368819460272789, + "num_tokens": 1282900.0, + "reward": 1.6600000619888307, + "reward_std": 1.03403559923172, + "rewards/boxed_rate/mean": 0.7666666626930236, + "rewards/boxed_rate/std": 0.3761233925819397, + "rewards/correctness/mean": 0.33333333432674406, + "rewards/correctness/std": 0.4298781991004944, + "rewards/multi_component_reward/mean": 0.3266666799783707, + "rewards/multi_component_reward/std": 0.6201063513755798, + "rewards/no_answer_rate/mean": 0.2333333373069763, + "rewards/no_answer_rate/std": 0.3761233925819397, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.24176665293052793, + "sampling/importance_sampling_ratio/mean": 0.05982898166403174, + "sampling/importance_sampling_ratio/min": 0.010627524812724437, + "sampling/sampling_logp_difference/max": 0.3079464197158813, + "sampling/sampling_logp_difference/mean": 0.009482560632750391, + "step": 195, + "step_time": 43.80696959905326 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1874.4, + "completions/max_terminated_length": 1207.6, + "completions/mean_length": 1102.2666748046875, + "completions/mean_terminated_length": 956.0200134277344, + "completions/min_length": 668.8, + "completions/min_terminated_length": 668.8, + "entropy": 0.13664823652555544, + "epoch": 0.07285974499089254, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01865367591381073, + "kl": 0.00029515944139954324, + "learning_rate": 4.881134969325153e-07, + "loss": 0.006076012551784515, + "num_tokens": 1318920.0, + "reward": 1.9533334493637085, + "reward_std": 1.1985347509384154, + "rewards/boxed_rate/mean": 0.8666666507720947, + "rewards/boxed_rate/std": 0.3265986442565918, + "rewards/correctness/mean": 0.4333333373069763, + "rewards/correctness/std": 0.5351926565170289, + "rewards/multi_component_reward/mean": 0.4866666853427887, + "rewards/multi_component_reward/std": 0.7141769766807556, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.3265986442565918, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.08207609243690968, + "sampling/importance_sampling_ratio/mean": 0.03297081319615245, + "sampling/importance_sampling_ratio/min": 0.003100475773135258, + "sampling/sampling_logp_difference/max": 0.40416555404663085, + "sampling/sampling_logp_difference/mean": 0.008233289048075677, + "step": 200, + "step_time": 42.36675942577422 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1600.0, + "completions/max_terminated_length": 1066.6, + "completions/mean_length": 959.2000122070312, + "completions/mean_terminated_length": 798.0866943359375, + "completions/min_length": 461.0, + "completions/min_terminated_length": 461.0, + "entropy": 0.14408907170097032, + "epoch": 0.07468123861566485, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.032731793820858, + "kl": 0.0003691373499653613, + "learning_rate": 4.871549079754601e-07, + "loss": -0.001960751600563526, + "num_tokens": 1352016.0, + "reward": 1.673333430290222, + "reward_std": 0.9306211471557617, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.3000000029802322, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.3400000214576721, + "rewards/multi_component_reward/std": 0.5803858906030654, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.10455413907766342, + "sampling/importance_sampling_ratio/mean": 0.03612115085124969, + "sampling/importance_sampling_ratio/min": 0.0026871272621811215, + "sampling/sampling_logp_difference/max": 0.3213069200515747, + "sampling/sampling_logp_difference/mean": 0.00922326734289527, + "step": 205, + "step_time": 37.272668573819104 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1572.2, + "completions/max_terminated_length": 1133.2, + "completions/mean_length": 879.3333618164063, + "completions/mean_terminated_length": 743.8000244140625, + "completions/min_length": 459.6, + "completions/min_terminated_length": 459.6, + "entropy": 0.19108737086256344, + "epoch": 0.07650273224043716, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018986158072948456, + "kl": 0.0004163069315836765, + "learning_rate": 4.86196319018405e-07, + "loss": 0.0035621844232082366, + "num_tokens": 1380880.0, + "reward": 1.740000104904175, + "reward_std": 0.947225558757782, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.33333333432674406, + "rewards/correctness/std": 0.4298781991004944, + "rewards/multi_component_reward/mean": 0.40666669309139253, + "rewards/multi_component_reward/std": 0.5280151665210724, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07124827802181244, + "sampling/importance_sampling_ratio/mean": 0.02541533587500453, + "sampling/importance_sampling_ratio/min": 0.002891992630936181, + "sampling/sampling_logp_difference/max": 0.3407265663146973, + "sampling/sampling_logp_difference/mean": 0.012478712201118469, + "step": 210, + "step_time": 35.72363304812461 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1773.4, + "completions/max_terminated_length": 1327.6, + "completions/mean_length": 1154.46669921875, + "completions/mean_terminated_length": 950.5800048828125, + "completions/min_length": 652.4, + "completions/min_terminated_length": 652.4, + "entropy": 0.18063361259798208, + "epoch": 0.07832422586520947, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.02516000345349312, + "kl": 0.0003309295129535409, + "learning_rate": 4.852377300613496e-07, + "loss": -0.003862759843468666, + "num_tokens": 1418892.0, + "reward": 1.766666793823242, + "reward_std": 0.897074180841446, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.33333333432674406, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.3333333648741245, + "rewards/multi_component_reward/std": 0.5403176128864289, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.24494898319244385, + "sampling/importance_sampling_ratio/max": 0.04162683514878154, + "sampling/importance_sampling_ratio/mean": 0.011416873242706061, + "sampling/importance_sampling_ratio/min": 0.001456704799248534, + "sampling/sampling_logp_difference/max": 0.3141038179397583, + "sampling/sampling_logp_difference/mean": 0.011231625638902187, + "step": 215, + "step_time": 40.99782377649099 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1087.4, + "completions/max_terminated_length": 1005.0, + "completions/mean_length": 687.36669921875, + "completions/mean_terminated_length": 652.033349609375, + "completions/min_length": 441.2, + "completions/min_terminated_length": 441.2, + "entropy": 0.20975149522225062, + "epoch": 0.08014571948998178, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.026713233441114426, + "kl": 0.0004683414850054154, + "learning_rate": 4.842791411042944e-07, + "loss": -0.0019101161509752274, + "num_tokens": 1442585.0, + "reward": 1.9400001287460327, + "reward_std": 1.0250610709190369, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.4, + "rewards/correctness/std": 0.45777305364608767, + "rewards/multi_component_reward/mean": 0.5066666960716247, + "rewards/multi_component_reward/std": 0.49611697196960447, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.11259435554966331, + "sampling/importance_sampling_ratio/mean": 0.04658988020382822, + "sampling/importance_sampling_ratio/min": 0.0064018826728026255, + "sampling/sampling_logp_difference/max": 0.35799312591552734, + "sampling/sampling_logp_difference/mean": 0.013789117708802224, + "step": 220, + "step_time": 25.938872035220264 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1644.6, + "completions/max_terminated_length": 1458.8, + "completions/mean_length": 1090.4333618164062, + "completions/mean_terminated_length": 1024.5933349609375, + "completions/min_length": 586.4, + "completions/min_terminated_length": 586.4, + "entropy": 0.15132917563120524, + "epoch": 0.08196721311475409, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.015617283061146736, + "kl": 0.00032217044063145297, + "learning_rate": 4.833205521472393e-07, + "loss": 0.005397457629442215, + "num_tokens": 1478730.0, + "reward": 1.2733334183692933, + "reward_std": 0.49739299416542054, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.10000000298023223, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.14000000953674316, + "rewards/multi_component_reward/std": 0.3222720980644226, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.09620626708492637, + "sampling/importance_sampling_ratio/mean": 0.030177546793129294, + "sampling/importance_sampling_ratio/min": 0.0006849504668480222, + "sampling/sampling_logp_difference/max": 0.35564944744110105, + "sampling/sampling_logp_difference/mean": 0.009922309406101703, + "step": 225, + "step_time": 38.19373752269894 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.26666667461395266, + "completions/max_length": 1435.0, + "completions/max_terminated_length": 1110.6, + "completions/mean_length": 1050.7667114257813, + "completions/mean_terminated_length": 875.8666809082031, + "completions/min_length": 714.8, + "completions/min_terminated_length": 714.8, + "entropy": 0.20316853250066438, + "epoch": 0.08378870673952642, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.021429063752293587, + "kl": 0.00040220142885421715, + "learning_rate": 4.82361963190184e-07, + "loss": -0.00029108244925737383, + "num_tokens": 1513091.0, + "reward": 1.3133334159851073, + "reward_std": 0.7383940577507019, + "rewards/boxed_rate/mean": 0.7333333343267441, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.11333334147930145, + "rewards/multi_component_reward/std": 0.4417478919029236, + "rewards/no_answer_rate/mean": 0.2666666626930237, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.14741267394274474, + "sampling/importance_sampling_ratio/mean": 0.06669870759360493, + "sampling/importance_sampling_ratio/min": 0.0190697070662623, + "sampling/sampling_logp_difference/max": 0.34659633636474607, + "sampling/sampling_logp_difference/mean": 0.013104908354580402, + "step": 230, + "step_time": 33.9486582595855 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1370.2, + "completions/max_terminated_length": 1313.4, + "completions/mean_length": 1031.9666870117187, + "completions/mean_terminated_length": 1006.8400268554688, + "completions/min_length": 716.6, + "completions/min_terminated_length": 716.6, + "entropy": 0.14171662541727226, + "epoch": 0.08561020036429873, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.031329505145549774, + "kl": 0.0002987771406575727, + "learning_rate": 4.814033742331288e-07, + "loss": -0.0023104714229702948, + "num_tokens": 1546912.0, + "reward": 1.7100001096725463, + "reward_std": 0.7802696824073792, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.3, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.41000002026557925, + "rewards/multi_component_reward/std": 0.43204120099544524, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.06215879218652844, + "sampling/importance_sampling_ratio/mean": 0.030630824994295834, + "sampling/importance_sampling_ratio/min": 0.007751126746109232, + "sampling/sampling_logp_difference/max": 0.3309502720832825, + "sampling/sampling_logp_difference/mean": 0.009595037158578634, + "step": 235, + "step_time": 33.03153369966894 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.3000000029802322, + "completions/max_length": 1976.2, + "completions/max_terminated_length": 1429.4, + "completions/mean_length": 1217.8667114257812, + "completions/mean_terminated_length": 870.3533569335938, + "completions/min_length": 525.4, + "completions/min_terminated_length": 525.4, + "entropy": 0.1393390517681837, + "epoch": 0.08743169398907104, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05459148809313774, + "kl": 0.0003685045870952308, + "learning_rate": 4.804447852760736e-07, + "loss": -0.0035786613821983337, + "num_tokens": 1587708.0, + "reward": 1.52666677236557, + "reward_std": 0.9796114563941956, + "rewards/boxed_rate/mean": 0.7, + "rewards/boxed_rate/std": 0.404018247127533, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.3823883533477783, + "rewards/multi_component_reward/mean": 0.19333335161209106, + "rewards/multi_component_reward/std": 0.6527607858180999, + "rewards/no_answer_rate/mean": 0.3000000029802322, + "rewards/no_answer_rate/std": 0.404018247127533, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.08990185633301735, + "sampling/importance_sampling_ratio/mean": 0.030038297548890112, + "sampling/importance_sampling_ratio/min": 0.00043211893236647645, + "sampling/sampling_logp_difference/max": 0.38284850120544434, + "sampling/sampling_logp_difference/mean": 0.008784445654600858, + "step": 240, + "step_time": 45.60475273691118 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1608.4, + "completions/max_terminated_length": 1074.2, + "completions/mean_length": 920.5666870117187, + "completions/mean_terminated_length": 754.0433471679687, + "completions/min_length": 548.6, + "completions/min_terminated_length": 548.6, + "entropy": 0.13041699826717376, + "epoch": 0.08925318761384335, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.011656666174530983, + "kl": 0.00033540410368004814, + "learning_rate": 4.794861963190184e-07, + "loss": -0.004278642311692238, + "num_tokens": 1618097.0, + "reward": 1.3933334350585938, + "reward_std": 0.31584063470363616, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.10327955484390258, + "rewards/multi_component_reward/mean": 0.1266666680574417, + "rewards/multi_component_reward/std": 0.3364715576171875, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.13333333730697633, + "rewards/repetition_rate/std": 0.2665788769721985, + "sampling/importance_sampling_ratio/max": 0.13740080818533898, + "sampling/importance_sampling_ratio/mean": 0.05796990524977445, + "sampling/importance_sampling_ratio/min": 0.006626390311389762, + "sampling/sampling_logp_difference/max": 0.3156670331954956, + "sampling/sampling_logp_difference/mean": 0.008615392539650202, + "step": 245, + "step_time": 36.637498759664595 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1533.4, + "completions/max_terminated_length": 1395.2, + "completions/mean_length": 973.1333923339844, + "completions/mean_terminated_length": 939.9467224121094, + "completions/min_length": 659.4, + "completions/min_terminated_length": 659.4, + "entropy": 0.14471415306131044, + "epoch": 0.09107468123861566, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.012578152120113373, + "kl": 0.00044469132699305193, + "learning_rate": 4.785276073619632e-07, + "loss": 0.00015797601081430913, + "num_tokens": 1651371.0, + "reward": 2.2233334302902223, + "reward_std": 0.6588261000812053, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.5333333432674408, + "rewards/correctness/std": 0.3098386645317078, + "rewards/multi_component_reward/mean": 0.656666674464941, + "rewards/multi_component_reward/std": 0.41430723667144775, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.18325970163568855, + "sampling/importance_sampling_ratio/mean": 0.10900020101107658, + "sampling/importance_sampling_ratio/min": 0.058780076946641203, + "sampling/sampling_logp_difference/max": 0.28016397953033445, + "sampling/sampling_logp_difference/mean": 0.009201886225491762, + "step": 250, + "step_time": 36.18357318136841 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1565.8, + "completions/max_terminated_length": 1137.4, + "completions/mean_length": 961.133349609375, + "completions/mean_terminated_length": 823.0366821289062, + "completions/min_length": 616.4, + "completions/min_terminated_length": 616.4, + "entropy": 0.14495401444534461, + "epoch": 0.09289617486338798, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.010682695545256138, + "kl": 0.00035484658534793806, + "learning_rate": 4.775690184049079e-07, + "loss": 0.0014964478090405465, + "num_tokens": 1682425.0, + "reward": 1.8333334445953369, + "reward_std": 0.6724297642707825, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.3666666716337204, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.4333333522081375, + "rewards/multi_component_reward/std": 0.4199430406093597, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.10162204280495643, + "sampling/importance_sampling_ratio/mean": 0.033536690101027486, + "sampling/importance_sampling_ratio/min": 0.00426993980855085, + "sampling/sampling_logp_difference/max": 0.33833720684051516, + "sampling/sampling_logp_difference/mean": 0.009549971111118794, + "step": 255, + "step_time": 35.81306515969336 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.4000000089406967, + "completions/max_length": 2048.0, + "completions/max_terminated_length": 1168.6, + "completions/mean_length": 1366.4000366210937, + "completions/mean_terminated_length": 998.9466796875, + "completions/min_length": 838.8, + "completions/min_terminated_length": 838.8, + "entropy": 0.14179731508096058, + "epoch": 0.0947176684881603, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05714869126677513, + "kl": 0.00032895591721171513, + "learning_rate": 4.7661042944785273e-07, + "loss": -0.004640129953622818, + "num_tokens": 1726081.0, + "reward": 1.4900000929832458, + "reward_std": 0.996655923128128, + "rewards/boxed_rate/mean": 0.6000000059604644, + "rewards/boxed_rate/std": 0.5010328412055969, + "rewards/correctness/mean": 0.2666666716337204, + "rewards/correctness/std": 0.3977532863616943, + "rewards/multi_component_reward/mean": 0.12333334572613239, + "rewards/multi_component_reward/std": 0.6989697873592376, + "rewards/no_answer_rate/mean": 0.4000000089406967, + "rewards/no_answer_rate/std": 0.5010328412055969, + "rewards/repetition_rate/mean": 0.1, + "rewards/repetition_rate/std": 0.10954451560974121, + "sampling/importance_sampling_ratio/max": 0.043305744789540766, + "sampling/importance_sampling_ratio/mean": 0.01504859896376729, + "sampling/importance_sampling_ratio/min": 4.04306270474383e-05, + "sampling/sampling_logp_difference/max": 0.3569210052490234, + "sampling/sampling_logp_difference/mean": 0.009369691275060177, + "step": 260, + "step_time": 46.90199479162693 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2666666716337204, + "completions/max_length": 1583.0, + "completions/max_terminated_length": 934.6, + "completions/mean_length": 1023.133349609375, + "completions/mean_terminated_length": 685.6266723632813, + "completions/min_length": 537.4, + "completions/min_terminated_length": 537.4, + "entropy": 0.17780156135559083, + "epoch": 0.0965391621129326, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018404532223939896, + "kl": 0.0003682690895705794, + "learning_rate": 4.756518404907975e-07, + "loss": 0.012411002814769746, + "num_tokens": 1759811.0, + "reward": 1.3833333849906921, + "reward_std": 0.7943651616573334, + "rewards/boxed_rate/mean": 0.7333333313465118, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.15000001043081285, + "rewards/multi_component_reward/std": 0.5146282583475112, + "rewards/no_answer_rate/mean": 0.2666666716337204, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.1577057947986759, + "sampling/importance_sampling_ratio/mean": 0.09503802299732342, + "sampling/importance_sampling_ratio/min": 0.0009112325180752826, + "sampling/sampling_logp_difference/max": 0.3926606416702271, + "sampling/sampling_logp_difference/mean": 0.012097407877445222, + "step": 265, + "step_time": 36.53071039505303 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1533.2, + "completions/max_terminated_length": 932.6, + "completions/mean_length": 900.7333618164063, + "completions/mean_terminated_length": 726.6700073242188, + "completions/min_length": 482.4, + "completions/min_terminated_length": 482.4, + "entropy": 0.18941813930869103, + "epoch": 0.09836065573770492, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.1255546510219574, + "kl": 0.0004568617567808057, + "learning_rate": 4.746932515337423e-07, + "loss": -0.006510256975889206, + "num_tokens": 1790271.0, + "reward": 1.7666667699813843, + "reward_std": 1.1080477476119994, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.33333333730697634, + "rewards/correctness/std": 0.48566790819168093, + "rewards/multi_component_reward/mean": 0.33333334922790525, + "rewards/multi_component_reward/std": 0.6733725190162658, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.18492921590805053, + "sampling/importance_sampling_ratio/max": 0.0878407845273614, + "sampling/importance_sampling_ratio/mean": 0.026912105677183717, + "sampling/importance_sampling_ratio/min": 0.0006732676178655298, + "sampling/sampling_logp_difference/max": 0.33626770973205566, + "sampling/sampling_logp_difference/mean": 0.012423686124384404, + "step": 270, + "step_time": 35.3790395591408 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1538.0, + "completions/max_terminated_length": 1314.2, + "completions/mean_length": 987.2333618164063, + "completions/mean_terminated_length": 886.1933471679688, + "completions/min_length": 572.4, + "completions/min_terminated_length": 572.4, + "entropy": 0.17079306667049726, + "epoch": 0.10018214936247723, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.0078973937779665, + "kl": 0.00039174315752461554, + "learning_rate": 4.737346625766871e-07, + "loss": -0.0023402150720357893, + "num_tokens": 1822306.0, + "reward": 1.833333396911621, + "reward_std": 0.9216741323471069, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.3666666716337204, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.43333334028720855, + "rewards/multi_component_reward/std": 0.5590049713850022, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.05113908690400422, + "sampling/importance_sampling_ratio/mean": 0.021760703669860958, + "sampling/importance_sampling_ratio/min": 0.008651768978314297, + "sampling/sampling_logp_difference/max": 0.4047112584114075, + "sampling/sampling_logp_difference/mean": 0.01078464426100254, + "step": 275, + "step_time": 35.81637797001749 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1824.0, + "completions/max_terminated_length": 1109.4, + "completions/mean_length": 981.833349609375, + "completions/mean_terminated_length": 751.3400024414062, + "completions/min_length": 517.2, + "completions/min_terminated_length": 517.2, + "entropy": 0.15676350990931193, + "epoch": 0.10200364298724955, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01452692411839962, + "kl": 0.0003485525502280022, + "learning_rate": 4.727760736196319e-07, + "loss": -0.004090853407979012, + "num_tokens": 1855067.0, + "reward": 1.4933334350585938, + "reward_std": 0.7522169463336468, + "rewards/boxed_rate/mean": 0.799999988079071, + "rewards/boxed_rate/std": 0.35449349880218506, + "rewards/correctness/mean": 0.23333333432674408, + "rewards/correctness/std": 0.30073869228363037, + "rewards/multi_component_reward/mean": 0.2266666792333126, + "rewards/multi_component_reward/std": 0.5248861283063888, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.35449349880218506, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12218071203678846, + "sampling/importance_sampling_ratio/mean": 0.04405408292077482, + "sampling/importance_sampling_ratio/min": 0.00501076535471764, + "sampling/sampling_logp_difference/max": 0.37753498554229736, + "sampling/sampling_logp_difference/mean": 0.009699719026684761, + "step": 280, + "step_time": 40.97467570956796 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1744.4, + "completions/max_terminated_length": 1173.6, + "completions/mean_length": 970.86669921875, + "completions/mean_terminated_length": 758.7533447265625, + "completions/min_length": 479.4, + "completions/min_terminated_length": 479.4, + "entropy": 0.15489849572380385, + "epoch": 0.10382513661202186, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.09483744204044342, + "kl": 0.000336567038417949, + "learning_rate": 4.718174846625766e-07, + "loss": -0.004700837284326553, + "num_tokens": 1886869.0, + "reward": 1.7000001311302184, + "reward_std": 0.8241950333118438, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.33333333730697634, + "rewards/correctness/std": 0.32236858606338503, + "rewards/multi_component_reward/mean": 0.36666667461395264, + "rewards/multi_component_reward/std": 0.5094490587711334, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.13168737590312957, + "sampling/importance_sampling_ratio/mean": 0.06171476859599352, + "sampling/importance_sampling_ratio/min": 0.010470659041346208, + "sampling/sampling_logp_difference/max": 0.344563889503479, + "sampling/sampling_logp_difference/mean": 0.010558286216109991, + "step": 285, + "step_time": 39.89412596244365 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333333432674408, + "completions/max_length": 1809.2, + "completions/max_terminated_length": 1132.6, + "completions/mean_length": 1119.7000366210937, + "completions/mean_terminated_length": 832.9400268554688, + "completions/min_length": 590.4, + "completions/min_terminated_length": 590.4, + "entropy": 0.15704507902264594, + "epoch": 0.10564663023679417, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.023737642914056778, + "kl": 0.0003448050100511561, + "learning_rate": 4.7085889570552147e-07, + "loss": 0.0019955366849899294, + "num_tokens": 1923040.0, + "reward": 1.380000114440918, + "reward_std": 0.5739886239171028, + "rewards/boxed_rate/mean": 0.7666666626930236, + "rewards/boxed_rate/std": 0.30073869228363037, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2128240704536438, + "rewards/multi_component_reward/mean": 0.11333334371447563, + "rewards/multi_component_reward/std": 0.46365060210227965, + "rewards/no_answer_rate/mean": 0.23333333432674408, + "rewards/no_answer_rate/std": 0.30073869228363037, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.18492921590805053, + "sampling/importance_sampling_ratio/max": 0.04411560408771038, + "sampling/importance_sampling_ratio/mean": 0.01753278383985162, + "sampling/importance_sampling_ratio/min": 0.0009861058487468544, + "sampling/sampling_logp_difference/max": 0.3644953012466431, + "sampling/sampling_logp_difference/mean": 0.010005151480436325, + "step": 290, + "step_time": 41.05979204457253 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1592.8, + "completions/max_terminated_length": 1143.4, + "completions/mean_length": 987.1000366210938, + "completions/mean_terminated_length": 825.9633544921875, + "completions/min_length": 591.8, + "completions/min_terminated_length": 591.8, + "entropy": 0.16288827657699584, + "epoch": 0.10746812386156648, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.026686564087867737, + "kl": 0.0003618090704549104, + "learning_rate": 4.6990030674846625e-07, + "loss": -0.0018155150115489959, + "num_tokens": 1955647.0, + "reward": 1.7666668176651001, + "reward_std": 0.4796482801437378, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.33333333432674406, + "rewards/correctness/std": 0.19119417667388916, + "rewards/multi_component_reward/mean": 0.36666668206453323, + "rewards/multi_component_reward/std": 0.34831122159957884, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.17200291641056537, + "sampling/importance_sampling_ratio/mean": 0.09221775899641216, + "sampling/importance_sampling_ratio/min": 0.053504807552870126, + "sampling/sampling_logp_difference/max": 0.31802849769592284, + "sampling/sampling_logp_difference/mean": 0.010488973837345839, + "step": 295, + "step_time": 36.921972643770275 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1181.2, + "completions/max_terminated_length": 1068.4, + "completions/mean_length": 714.5666870117187, + "completions/mean_terminated_length": 675.57333984375, + "completions/min_length": 433.8, + "completions/min_terminated_length": 433.8, + "entropy": 0.20909111325939497, + "epoch": 0.1092896174863388, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.9235589504241943, + "kl": 0.0007714893268712331, + "learning_rate": 4.68941717791411e-07, + "loss": 0.0028855174779891966, + "num_tokens": 1980240.0, + "reward": 1.8966667413711549, + "reward_std": 1.0613920927047729, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.3666666716337204, + "rewards/correctness/std": 0.5072978019714356, + "rewards/multi_component_reward/mean": 0.4633333578705788, + "rewards/multi_component_reward/std": 0.5980847001075744, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.11060594655573368, + "sampling/importance_sampling_ratio/mean": 0.05734073922503739, + "sampling/importance_sampling_ratio/min": 0.024227803308775008, + "sampling/sampling_logp_difference/max": 0.31456995010375977, + "sampling/sampling_logp_difference/mean": 0.013103757984936237, + "step": 300, + "step_time": 27.952531585469842 + }, + { + "epoch": 0.1092896174863388, + "eval_clip_ratio/high_max": 0.0, + "eval_clip_ratio/high_mean": 0.0, + "eval_clip_ratio/low_mean": 0.0, + "eval_clip_ratio/low_min": 0.0, + "eval_clip_ratio/region_mean": 0.0, + "eval_completions/clipped_ratio": 0.22666667222976686, + "eval_completions/max_length": 1758.54, + "eval_completions/max_terminated_length": 1125.4, + "eval_completions/mean_length": 1107.770029296875, + "eval_completions/mean_terminated_length": 846.1800146484375, + "eval_completions/min_length": 606.8, + "eval_completions/min_terminated_length": 606.8, + "eval_entropy": 0.17873862124979495, + "eval_frac_reward_zero_std": 0.08, + "eval_kl": 0.0003903971218096558, + "eval_loss": -0.0051070391200482845, + "eval_num_tokens": 1980240.0, + "eval_reward": 1.5980000698566437, + "eval_reward_std": 0.782314371317625, + "eval_rewards/boxed_rate/mean": 0.7766666629910469, + "eval_rewards/boxed_rate/std": 0.33167909026145936, + "eval_rewards/correctness/mean": 0.28666666984558103, + "eval_rewards/correctness/std": 0.30271870851516725, + "eval_rewards/multi_component_reward/mean": 0.26800001293420794, + "eval_rewards/multi_component_reward/std": 0.5066750717163085, + "eval_rewards/no_answer_rate/mean": 0.22333333760499954, + "eval_rewards/no_answer_rate/std": 0.33167909026145936, + "eval_rewards/repetition_rate/mean": 0.04333333432674408, + "eval_rewards/repetition_rate/std": 0.08060015916824341, + "eval_runtime": 1753.4069, + "eval_samples_per_second": 0.029, + "eval_sampling/importance_sampling_ratio/max": 0.0897229278186569, + "eval_sampling/importance_sampling_ratio/mean": 0.035299012192626836, + "eval_sampling/importance_sampling_ratio/min": 0.0025020323396462286, + "eval_sampling/sampling_logp_difference/max": 0.5019185048341751, + "eval_sampling/sampling_logp_difference/mean": 0.012026464603841304, + "eval_steps_per_second": 0.005, + "step": 300 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1, + "completions/max_length": 1605.6, + "completions/max_terminated_length": 1384.6, + "completions/mean_length": 1010.7000366210938, + "completions/mean_terminated_length": 876.26669921875, + "completions/min_length": 573.8, + "completions/min_terminated_length": 573.8, + "entropy": 0.1340769293407599, + "epoch": 0.1111111111111111, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.034374091774225235, + "kl": 0.000321884588629473, + "learning_rate": 4.6798312883435583e-07, + "loss": -0.014604152739048004, + "num_tokens": 2014467.0, + "reward": 2.043333411216736, + "reward_std": 0.8047043204307556, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.10954451560974121, + "rewards/correctness/mean": 0.4666666716337204, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.5433333471417428, + "rewards/multi_component_reward/std": 0.4605102062225342, + "rewards/no_answer_rate/mean": 0.1, + "rewards/no_answer_rate/std": 0.10954451560974121, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12428842782974243, + "sampling/importance_sampling_ratio/mean": 0.046624291501939294, + "sampling/importance_sampling_ratio/min": 0.0006754371514054114, + "sampling/sampling_logp_difference/max": 0.32104220390319826, + "sampling/sampling_logp_difference/mean": 0.008986939676105976, + "step": 305, + "step_time": 36.810268136300145 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1557.0, + "completions/max_terminated_length": 1291.8, + "completions/mean_length": 1068.033349609375, + "completions/mean_terminated_length": 916.2, + "completions/min_length": 631.6, + "completions/min_terminated_length": 631.6, + "entropy": 0.15544118496278922, + "epoch": 0.11293260473588343, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01212399173527956, + "kl": 0.00047503276291536165, + "learning_rate": 4.670245398773006e-07, + "loss": -8.383383974432945e-05, + "num_tokens": 2049616.0, + "reward": 1.6533334255218506, + "reward_std": 1.070458745956421, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2128240704536438, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.4794029474258423, + "rewards/multi_component_reward/mean": 0.3200000122189522, + "rewards/multi_component_reward/std": 0.6258516371250152, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2128240704536438, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.08939264337532223, + "sampling/importance_sampling_ratio/mean": 0.030286337621510027, + "sampling/importance_sampling_ratio/min": 0.0005470938450967466, + "sampling/sampling_logp_difference/max": 0.34738160371780397, + "sampling/sampling_logp_difference/mean": 0.010354180075228214, + "step": 310, + "step_time": 36.12143337074667 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.3333333432674408, + "completions/max_length": 1634.2, + "completions/max_terminated_length": 1091.4, + "completions/mean_length": 1218.6333618164062, + "completions/mean_terminated_length": 858.4666870117187, + "completions/min_length": 677.6, + "completions/min_terminated_length": 677.6, + "entropy": 0.17260866140325865, + "epoch": 0.11475409836065574, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05682944133877754, + "kl": 0.0005160604792763479, + "learning_rate": 4.6606595092024536e-07, + "loss": -0.006619427353143692, + "num_tokens": 2088659.0, + "reward": 2.090000092983246, + "reward_std": 1.1096696853637695, + "rewards/boxed_rate/mean": 0.6666666716337204, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.566666665673256, + "rewards/correctness/std": 0.451508092880249, + "rewards/multi_component_reward/mean": 0.523333391547203, + "rewards/multi_component_reward/std": 0.6607022762298584, + "rewards/no_answer_rate/mean": 0.33333333134651183, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.05110827875323594, + "sampling/importance_sampling_ratio/mean": 0.018865692079998553, + "sampling/importance_sampling_ratio/min": 0.0009169460441439363, + "sampling/sampling_logp_difference/max": 0.3731674671173096, + "sampling/sampling_logp_difference/mean": 0.01166527420282364, + "step": 315, + "step_time": 38.15295873656869 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1484.2, + "completions/max_terminated_length": 1039.2, + "completions/mean_length": 931.7000244140625, + "completions/mean_terminated_length": 790.5233520507812, + "completions/min_length": 563.4, + "completions/min_terminated_length": 563.4, + "entropy": 0.143128818521897, + "epoch": 0.11657559198542805, + "frac_reward_zero_std": 0.6, + "grad_norm": 0.005422326270490885, + "kl": 0.0004371852917150439, + "learning_rate": 4.6510736196319015e-07, + "loss": 0.004143273830413819, + "num_tokens": 2119250.0, + "reward": 1.5300001502037048, + "reward_std": 0.24947773814201354, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.23333333432674408, + "rewards/correctness/std": 0.08164966106414795, + "rewards/multi_component_reward/mean": 0.29666668176651, + "rewards/multi_component_reward/std": 0.17254199385643004, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.12071083066985011, + "sampling/importance_sampling_ratio/mean": 0.05540300235152244, + "sampling/importance_sampling_ratio/min": 0.013269457093102949, + "sampling/sampling_logp_difference/max": 0.36359539031982424, + "sampling/sampling_logp_difference/mean": 0.009230617992579937, + "step": 320, + "step_time": 34.49489349294454 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1798.0, + "completions/max_terminated_length": 1537.6, + "completions/mean_length": 1144.8333618164063, + "completions/mean_terminated_length": 965.6733520507812, + "completions/min_length": 634.0, + "completions/min_terminated_length": 634.0, + "entropy": 0.18388984675208728, + "epoch": 0.11839708561020036, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.14283621311187744, + "kl": 0.0006492346903542057, + "learning_rate": 4.64148773006135e-07, + "loss": -0.0025676295161247255, + "num_tokens": 2157267.0, + "reward": 1.630000066757202, + "reward_std": 0.89271479845047, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.3761233925819397, + "rewards/multi_component_reward/mean": 0.3300000101327896, + "rewards/multi_component_reward/std": 0.5265865564346314, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.04229360092431307, + "sampling/importance_sampling_ratio/mean": 0.019075372768566013, + "sampling/importance_sampling_ratio/min": 8.805350876674933e-05, + "sampling/sampling_logp_difference/max": 0.3247460603713989, + "sampling/sampling_logp_difference/mean": 0.01177409002557397, + "step": 325, + "step_time": 41.04225417692214 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 823.4, + "completions/max_terminated_length": 823.4, + "completions/mean_length": 544.1333435058593, + "completions/mean_terminated_length": 544.1333435058593, + "completions/min_length": 366.0, + "completions/min_terminated_length": 366.0, + "entropy": 0.22606497158606847, + "epoch": 0.12021857923497267, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.9119893908500671, + "kl": 0.0007021349136872838, + "learning_rate": 4.631901840490797e-07, + "loss": -0.006102682277560234, + "num_tokens": 2175259.0, + "reward": 1.8700001001358033, + "reward_std": 0.8484382510185242, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.3666666716337204, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.503333343565464, + "rewards/multi_component_reward/std": 0.44442008137702943, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.13480582600459456, + "sampling/importance_sampling_ratio/mean": 0.07863845487590879, + "sampling/importance_sampling_ratio/min": 0.01574344330747408, + "sampling/sampling_logp_difference/max": 0.3204054594039917, + "sampling/sampling_logp_difference/mean": 0.014066471531987191, + "step": 330, + "step_time": 20.059878361225127 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1395.2, + "completions/max_terminated_length": 910.6, + "completions/mean_length": 742.1333679199219, + "completions/mean_terminated_length": 640.480029296875, + "completions/min_length": 509.6, + "completions/min_terminated_length": 509.6, + "entropy": 0.2093804210424423, + "epoch": 0.122040072859745, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.028569765388965607, + "kl": 0.0006724427085525046, + "learning_rate": 4.622315950920245e-07, + "loss": -0.0013746816664934158, + "num_tokens": 2200067.0, + "reward": 1.6200001239776611, + "reward_std": 0.8776960492134094, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.3823883533477783, + "rewards/multi_component_reward/mean": 0.3533333420753479, + "rewards/multi_component_reward/std": 0.49811467826366423, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07016745954751968, + "sampling/importance_sampling_ratio/mean": 0.027400266751646994, + "sampling/importance_sampling_ratio/min": 0.0014501588469148616, + "sampling/sampling_logp_difference/max": 0.36046857833862306, + "sampling/sampling_logp_difference/mean": 0.014280413649976254, + "step": 335, + "step_time": 31.59101020414382 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 1939.2, + "completions/max_terminated_length": 988.8, + "completions/mean_length": 976.86669921875, + "completions/mean_terminated_length": 681.2466857910156, + "completions/min_length": 446.6, + "completions/min_terminated_length": 446.6, + "entropy": 0.2083836982647578, + "epoch": 0.12386156648451731, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.06303755939006805, + "kl": 0.0005712434819239812, + "learning_rate": 4.612730061349693e-07, + "loss": -0.0044234395027160645, + "num_tokens": 2232145.0, + "reward": 1.74333336353302, + "reward_std": 0.9833520889282227, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.33333334028720857, + "rewards/correctness/std": 0.3977532863616943, + "rewards/multi_component_reward/mean": 0.3433333486318588, + "rewards/multi_component_reward/std": 0.6024900764226914, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.1221102561801672, + "sampling/importance_sampling_ratio/mean": 0.029469438176602126, + "sampling/importance_sampling_ratio/min": 5.1707089343915565e-05, + "sampling/sampling_logp_difference/max": 0.3329290866851807, + "sampling/sampling_logp_difference/mean": 0.012941631115972995, + "step": 340, + "step_time": 43.22656154055149 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1, + "completions/max_length": 1163.2, + "completions/max_terminated_length": 1144.4, + "completions/mean_length": 778.86669921875, + "completions/mean_terminated_length": 688.9000244140625, + "completions/min_length": 431.8, + "completions/min_terminated_length": 431.8, + "entropy": 0.1657411351799965, + "epoch": 0.12568306010928962, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.1018831878900528, + "kl": 0.0005899470415897667, + "learning_rate": 4.603144171779141e-07, + "loss": -0.006323814392089844, + "num_tokens": 2259027.0, + "reward": 1.5300001144409179, + "reward_std": 0.9171831011772156, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.10954451560974121, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.4298781991004944, + "rewards/multi_component_reward/mean": 0.23000000715255736, + "rewards/multi_component_reward/std": 0.5341363847255707, + "rewards/no_answer_rate/mean": 0.1, + "rewards/no_answer_rate/std": 0.10954451560974121, + "rewards/repetition_rate/mean": 0.1, + "rewards/repetition_rate/std": 0.10954451560974121, + "sampling/importance_sampling_ratio/max": 0.12837026529014112, + "sampling/importance_sampling_ratio/mean": 0.05263339746743441, + "sampling/importance_sampling_ratio/min": 0.0009968833521444183, + "sampling/sampling_logp_difference/max": 0.31432443857192993, + "sampling/sampling_logp_difference/mean": 0.011136513762176036, + "step": 345, + "step_time": 27.575114846229553 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1772.4, + "completions/max_terminated_length": 1112.4, + "completions/mean_length": 983.0333618164062, + "completions/mean_terminated_length": 776.57333984375, + "completions/min_length": 516.6, + "completions/min_terminated_length": 516.6, + "entropy": 0.18533113847176233, + "epoch": 0.12750455373406194, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.1743193119764328, + "kl": 0.0007398304010469777, + "learning_rate": 4.593558282208589e-07, + "loss": 0.0035149652510881426, + "num_tokens": 2292724.0, + "reward": 1.5366667270660401, + "reward_std": 0.7906980156898499, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.2366666778922081, + "rewards/multi_component_reward/std": 0.52751624584198, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.16946832239627838, + "sampling/importance_sampling_ratio/mean": 0.04226875379681587, + "sampling/importance_sampling_ratio/min": 0.00029637051690457807, + "sampling/sampling_logp_difference/max": 0.34900493621826173, + "sampling/sampling_logp_difference/mean": 0.011652881279587746, + "step": 350, + "step_time": 40.440953285992144 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1092.8, + "completions/max_terminated_length": 992.2, + "completions/mean_length": 720.4666931152344, + "completions/mean_terminated_length": 686.5133483886718, + "completions/min_length": 467.2, + "completions/min_terminated_length": 467.2, + "entropy": 0.1794678675631682, + "epoch": 0.12932604735883424, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.020413219928741455, + "kl": 0.0006120980954923046, + "learning_rate": 4.583972392638036e-07, + "loss": -0.0014667985960841179, + "num_tokens": 2316852.0, + "reward": 2.060000109672546, + "reward_std": 0.7259637594223023, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.46666666865348816, + "rewards/correctness/std": 0.32236858606338503, + "rewards/multi_component_reward/mean": 0.59333336353302, + "rewards/multi_component_reward/std": 0.4035952419042587, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.14755254588089883, + "sampling/importance_sampling_ratio/mean": 0.07640535607933999, + "sampling/importance_sampling_ratio/min": 0.03205070712427443, + "sampling/sampling_logp_difference/max": 0.3078671216964722, + "sampling/sampling_logp_difference/mean": 0.011428364552557468, + "step": 355, + "step_time": 26.221613752283154 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1573.4, + "completions/max_terminated_length": 1007.8, + "completions/mean_length": 839.6000122070312, + "completions/mean_terminated_length": 670.27333984375, + "completions/min_length": 423.0, + "completions/min_terminated_length": 423.0, + "entropy": 0.2592687545965115, + "epoch": 0.13114754098360656, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.21311414241790771, + "kl": 0.0006711701275586772, + "learning_rate": 4.5743865030674845e-07, + "loss": -0.0009625215083360672, + "num_tokens": 2345424.0, + "reward": 1.6000000953674316, + "reward_std": 0.7632826209068299, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.2666666656732559, + "rewards/correctness/std": 0.3265986442565918, + "rewards/multi_component_reward/mean": 0.33333335518836976, + "rewards/multi_component_reward/std": 0.44263782203197477, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.06810822673141956, + "sampling/importance_sampling_ratio/mean": 0.026497452543117105, + "sampling/importance_sampling_ratio/min": 0.0005043454068991798, + "sampling/sampling_logp_difference/max": 0.46866347789764407, + "sampling/sampling_logp_difference/mean": 0.01661387998610735, + "step": 360, + "step_time": 35.86930139716715 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333334028720856, + "completions/max_length": 1906.6, + "completions/max_terminated_length": 1405.8, + "completions/mean_length": 1317.56669921875, + "completions/mean_terminated_length": 1097.8466796875, + "completions/min_length": 801.0, + "completions/min_terminated_length": 801.0, + "entropy": 0.11761416780451933, + "epoch": 0.13296903460837886, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.015462023206055164, + "kl": 0.0003710240945414019, + "learning_rate": 4.5648006134969324e-07, + "loss": -0.007586926966905594, + "num_tokens": 2388119.0, + "reward": 1.7300000667572022, + "reward_std": 1.0704036235809327, + "rewards/boxed_rate/mean": 0.7666666686534882, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.33333333730697634, + "rewards/correctness/std": 0.48566790819168093, + "rewards/multi_component_reward/mean": 0.2966666899621487, + "rewards/multi_component_reward/std": 0.6780295848846436, + "rewards/no_answer_rate/mean": 0.23333334028720856, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.24494898319244385, + "sampling/importance_sampling_ratio/max": 0.04545931427273899, + "sampling/importance_sampling_ratio/mean": 0.02320328297209926, + "sampling/importance_sampling_ratio/min": 0.001607318179527617, + "sampling/sampling_logp_difference/max": 0.35903769731521606, + "sampling/sampling_logp_difference/mean": 0.007444971334189176, + "step": 365, + "step_time": 43.916714335232975 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1418.0, + "completions/max_terminated_length": 1135.4, + "completions/mean_length": 861.5666931152343, + "completions/mean_terminated_length": 710.4833435058594, + "completions/min_length": 346.6, + "completions/min_terminated_length": 346.6, + "entropy": 0.22315610001484554, + "epoch": 0.13479052823315119, + "frac_reward_zero_std": 0.0, + "grad_norm": 1.7706270217895508, + "kl": 0.0008742218895349651, + "learning_rate": 4.55521472392638e-07, + "loss": 0.003686445206403732, + "num_tokens": 2416660.0, + "reward": 1.5800000548362731, + "reward_std": 0.8671060860157013, + "rewards/boxed_rate/mean": 0.8666666746139526, + "rewards/boxed_rate/std": 0.20655910968780516, + "rewards/correctness/mean": 0.26666667461395266, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.31333333253860474, + "rewards/multi_component_reward/std": 0.5032015323638916, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.20655910968780516, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.20161812007427216, + "sampling/importance_sampling_ratio/mean": 0.09829787239432335, + "sampling/importance_sampling_ratio/min": 0.056043115374653285, + "sampling/sampling_logp_difference/max": 0.35296247005462644, + "sampling/sampling_logp_difference/mean": 0.014671148918569087, + "step": 370, + "step_time": 33.350730242021385 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333432674407, + "completions/max_length": 1741.8, + "completions/max_terminated_length": 1278.2, + "completions/mean_length": 958.4000366210937, + "completions/mean_terminated_length": 774.8600219726562, + "completions/min_length": 536.0, + "completions/min_terminated_length": 536.0, + "entropy": 0.19409313425421715, + "epoch": 0.1366120218579235, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.10258854180574417, + "kl": 0.0005339390016160905, + "learning_rate": 4.545628834355828e-07, + "loss": 0.0006846859585493803, + "num_tokens": 2448526.0, + "reward": 1.6966667890548706, + "reward_std": 1.038489294052124, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.3000000059604645, + "rewards/correctness/std": 0.4794029474258423, + "rewards/multi_component_reward/mean": 0.3300000175833702, + "rewards/multi_component_reward/std": 0.6378163933753968, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.023186815530061723, + "sampling/importance_sampling_ratio/mean": 0.013102744333446026, + "sampling/importance_sampling_ratio/min": 0.00010366742491344506, + "sampling/sampling_logp_difference/max": 0.3774606704711914, + "sampling/sampling_logp_difference/mean": 0.011916628293693065, + "step": 375, + "step_time": 30.433830817975103 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333334028720856, + "completions/max_length": 1764.6, + "completions/max_terminated_length": 1185.0, + "completions/mean_length": 1108.233349609375, + "completions/mean_terminated_length": 801.3400024414062, + "completions/min_length": 541.8, + "completions/min_terminated_length": 541.8, + "entropy": 0.25079266702135405, + "epoch": 0.1384335154826958, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.015607405453920364, + "kl": 0.000638273500953801, + "learning_rate": 4.536042944785276e-07, + "loss": -0.004855351522564888, + "num_tokens": 2484623.0, + "reward": 1.6300000548362732, + "reward_std": 0.6760274231433868, + "rewards/boxed_rate/mean": 0.799999988079071, + "rewards/boxed_rate/std": 0.35449349880218506, + "rewards/correctness/mean": 0.2666666656732559, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.23000002317130566, + "rewards/multi_component_reward/std": 0.4806109189987183, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.35449349880218506, + "rewards/repetition_rate/mean": 0.13333333432674407, + "rewards/repetition_rate/std": 0.19119417667388916, + "sampling/importance_sampling_ratio/max": 0.030251298751682042, + "sampling/importance_sampling_ratio/mean": 0.01690196809358895, + "sampling/importance_sampling_ratio/min": 0.0027151843596084375, + "sampling/sampling_logp_difference/max": 0.3498866558074951, + "sampling/sampling_logp_difference/mean": 0.016159016452729703, + "step": 380, + "step_time": 15.216538714803756 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 1980.6, + "completions/max_terminated_length": 1102.2, + "completions/mean_length": 1102.2000122070312, + "completions/mean_terminated_length": 820.3700073242187, + "completions/min_length": 554.4, + "completions/min_terminated_length": 554.4, + "entropy": 0.21501451556881268, + "epoch": 0.14025500910746813, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.039810437709093094, + "kl": 0.0005428103109200795, + "learning_rate": 4.5264570552147234e-07, + "loss": -0.001014799065887928, + "num_tokens": 2521715.0, + "reward": 1.33000009059906, + "reward_std": 0.713744330406189, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.36985843181610106, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.16333335041999816, + "rewards/multi_component_reward/std": 0.4538270115852356, + "rewards/no_answer_rate/mean": 0.20000000596046447, + "rewards/no_answer_rate/std": 0.36985843181610106, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.050251276697963476, + "sampling/importance_sampling_ratio/mean": 0.014141232566908002, + "sampling/importance_sampling_ratio/min": 1.2361302467010191e-05, + "sampling/sampling_logp_difference/max": 0.30763993263244627, + "sampling/sampling_logp_difference/mean": 0.014410833641886712, + "step": 385, + "step_time": 16.672282354161144 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.33333333730697634, + "completions/max_length": 1734.4, + "completions/max_terminated_length": 1527.6, + "completions/mean_length": 1290.500048828125, + "completions/mean_terminated_length": 1057.36669921875, + "completions/min_length": 721.2, + "completions/min_terminated_length": 721.2, + "entropy": 0.17866969083746273, + "epoch": 0.14207650273224043, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.009094689041376114, + "kl": 0.0004922606613642226, + "learning_rate": 4.5168711656441713e-07, + "loss": -0.0016124360263347626, + "num_tokens": 2563196.0, + "reward": 1.5066667079925538, + "reward_std": 0.9714008927345276, + "rewards/boxed_rate/mean": 0.6666666686534881, + "rewards/boxed_rate/std": 0.32236858606338503, + "rewards/correctness/mean": 0.26666667461395266, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.17333333306014537, + "rewards/multi_component_reward/std": 0.564999234676361, + "rewards/no_answer_rate/mean": 0.33333333730697634, + "rewards/no_answer_rate/std": 0.32236858606338503, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.09146881643682718, + "sampling/importance_sampling_ratio/mean": 0.027744953380897643, + "sampling/importance_sampling_ratio/min": 0.0008299978789871564, + "sampling/sampling_logp_difference/max": 0.44667046070098876, + "sampling/sampling_logp_difference/mean": 0.011561266146600246, + "step": 390, + "step_time": 15.62863542791456 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1713.0, + "completions/max_terminated_length": 1077.4, + "completions/mean_length": 995.9333801269531, + "completions/mean_terminated_length": 786.106689453125, + "completions/min_length": 546.4, + "completions/min_terminated_length": 546.4, + "entropy": 0.1970411961277326, + "epoch": 0.14389799635701275, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018740270286798477, + "kl": 0.0007653967642302935, + "learning_rate": 4.50728527607362e-07, + "loss": 0.005758083239197731, + "num_tokens": 2595858.0, + "reward": 1.4666666984558105, + "reward_std": 0.6528747193515301, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.20000000223517417, + "rewards/multi_component_reward/std": 0.46670507490634916, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.09698612228967249, + "sampling/importance_sampling_ratio/mean": 0.030241910181939603, + "sampling/importance_sampling_ratio/min": 0.0021489314406773754, + "sampling/sampling_logp_difference/max": 0.34886856079101564, + "sampling/sampling_logp_difference/mean": 0.012738966010510921, + "step": 395, + "step_time": 14.849660068005324 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1575.0, + "completions/max_terminated_length": 948.8, + "completions/mean_length": 935.7000366210938, + "completions/mean_terminated_length": 702.1866821289062, + "completions/min_length": 482.0, + "completions/min_terminated_length": 482.0, + "entropy": 0.2041997093707323, + "epoch": 0.14571948998178508, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.10930619388818741, + "kl": 0.0005768983881959381, + "learning_rate": 4.497699386503067e-07, + "loss": -0.004532752931118012, + "num_tokens": 2626437.0, + "reward": 1.7466667890548706, + "reward_std": 0.8040140867233276, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.33333333134651183, + "rewards/correctness/std": 0.35449349880218506, + "rewards/multi_component_reward/mean": 0.34666669368743896, + "rewards/multi_component_reward/std": 0.5015990376472473, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.09340296909213067, + "sampling/importance_sampling_ratio/mean": 0.03166428431868553, + "sampling/importance_sampling_ratio/min": 0.007079391362589009, + "sampling/sampling_logp_difference/max": 0.4040722370147705, + "sampling/sampling_logp_difference/mean": 0.013007087912410498, + "step": 400, + "step_time": 13.86912056710571 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1673.6, + "completions/max_terminated_length": 1493.4, + "completions/mean_length": 1166.2000366210937, + "completions/mean_terminated_length": 1052.9500122070312, + "completions/min_length": 725.0, + "completions/min_terminated_length": 725.0, + "entropy": 0.1420179379483064, + "epoch": 0.14754098360655737, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.021460620686411858, + "kl": 0.0005037313070109425, + "learning_rate": 4.488113496932515e-07, + "loss": -0.0034239646047353745, + "num_tokens": 2665191.0, + "reward": 1.6500000953674316, + "reward_std": 0.5699952363967895, + "rewards/boxed_rate/mean": 0.8666666746139526, + "rewards/boxed_rate/std": 0.20655910968780516, + "rewards/correctness/mean": 0.2666666656732559, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.2833333633840084, + "rewards/multi_component_reward/std": 0.40620826482772826, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.20655910968780516, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.18492921590805053, + "sampling/importance_sampling_ratio/max": 0.047187263565137985, + "sampling/importance_sampling_ratio/mean": 0.016343344282358886, + "sampling/importance_sampling_ratio/min": 0.0005224805446630398, + "sampling/sampling_logp_difference/max": 0.33698827028274536, + "sampling/sampling_logp_difference/mean": 0.009091775678098202, + "step": 405, + "step_time": 14.932769652456045 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2333333373069763, + "completions/max_length": 1674.6, + "completions/max_terminated_length": 1293.0, + "completions/mean_length": 1194.5333679199218, + "completions/mean_terminated_length": 945.5333374023437, + "completions/min_length": 665.2, + "completions/min_terminated_length": 665.2, + "entropy": 0.19754986638824146, + "epoch": 0.1493624772313297, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.12479820102453232, + "kl": 0.000494376394393233, + "learning_rate": 4.4785276073619634e-07, + "loss": -0.0048382353037595745, + "num_tokens": 2707231.0, + "reward": 1.240000104904175, + "reward_std": 0.5429712414741517, + "rewards/boxed_rate/mean": 0.7666666746139527, + "rewards/boxed_rate/std": 0.3161036252975464, + "rewards/correctness/mean": 0.13333333432674407, + "rewards/correctness/std": 0.19119417667388916, + "rewards/multi_component_reward/mean": 0.10666667819023132, + "rewards/multi_component_reward/std": 0.35998966693878176, + "rewards/no_answer_rate/mean": 0.2333333373069763, + "rewards/no_answer_rate/std": 0.3161036252975464, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.10651108641177416, + "sampling/importance_sampling_ratio/mean": 0.03249359540641308, + "sampling/importance_sampling_ratio/min": 2.793159140067336e-05, + "sampling/sampling_logp_difference/max": 0.34277451038360596, + "sampling/sampling_logp_difference/mean": 0.01319133285433054, + "step": 410, + "step_time": 15.258313446864486 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 2048.0, + "completions/max_terminated_length": 1373.4, + "completions/mean_length": 1174.6666870117188, + "completions/mean_terminated_length": 982.5400207519531, + "completions/min_length": 717.8, + "completions/min_terminated_length": 717.8, + "entropy": 0.17395039709905782, + "epoch": 0.151183970856102, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.1571768969297409, + "kl": 0.0006294176423883376, + "learning_rate": 4.468941717791411e-07, + "loss": -0.013487279415130615, + "num_tokens": 2744805.0, + "reward": 1.6100000858306884, + "reward_std": 0.9736106634140015, + "rewards/boxed_rate/mean": 0.799999988079071, + "rewards/boxed_rate/std": 0.4298781991004944, + "rewards/correctness/mean": 0.3000000089406967, + "rewards/correctness/std": 0.3914883255958557, + "rewards/multi_component_reward/mean": 0.31000002175569535, + "rewards/multi_component_reward/std": 0.5971025705337525, + "rewards/no_answer_rate/mean": 0.20000000596046447, + "rewards/no_answer_rate/std": 0.4298781991004944, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.09501664619892836, + "sampling/importance_sampling_ratio/mean": 0.02351754056289792, + "sampling/importance_sampling_ratio/min": 1.3345446579433804e-05, + "sampling/sampling_logp_difference/max": 0.35430474281311036, + "sampling/sampling_logp_difference/mean": 0.012131126876920462, + "step": 415, + "step_time": 17.18821715861559 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1233.4, + "completions/max_terminated_length": 934.0, + "completions/mean_length": 838.5667053222656, + "completions/mean_terminated_length": 744.7300109863281, + "completions/min_length": 594.4, + "completions/min_terminated_length": 594.4, + "entropy": 0.17896714533368746, + "epoch": 0.15300546448087432, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.08533581346273422, + "kl": 0.00047911926230881363, + "learning_rate": 4.4593558282208586e-07, + "loss": -0.00437382236123085, + "num_tokens": 2771858.0, + "reward": 1.5300000667572022, + "reward_std": 0.8672845721244812, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.2333333373069763, + "rewards/correctness/std": 0.3761233925819397, + "rewards/multi_component_reward/mean": 0.29666668176651, + "rewards/multi_component_reward/std": 0.49711505472660067, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07356768772006035, + "sampling/importance_sampling_ratio/mean": 0.026540174335241317, + "sampling/importance_sampling_ratio/min": 0.0027435134909879364, + "sampling/sampling_logp_difference/max": 0.2972338914871216, + "sampling/sampling_logp_difference/mean": 0.011641639284789562, + "step": 420, + "step_time": 11.845613337121904 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1578.6, + "completions/max_terminated_length": 1112.6, + "completions/mean_length": 957.2333740234375, + "completions/mean_terminated_length": 789.5533447265625, + "completions/min_length": 506.8, + "completions/min_terminated_length": 506.8, + "entropy": 0.22455186421672504, + "epoch": 0.15482695810564662, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.011058821342885494, + "kl": 0.0005187941482290625, + "learning_rate": 4.449769938650307e-07, + "loss": -0.0018246347084641457, + "num_tokens": 2803527.0, + "reward": 1.6033333539962769, + "reward_std": 0.8526834607124328, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.26666667461395266, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.3033333346247673, + "rewards/multi_component_reward/std": 0.5273210048675537, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.040888981963507834, + "sampling/importance_sampling_ratio/mean": 0.012350313179194928, + "sampling/importance_sampling_ratio/min": 4.1122279969854155e-05, + "sampling/sampling_logp_difference/max": 0.3840524673461914, + "sampling/sampling_logp_difference/mean": 0.014351606741547585, + "step": 425, + "step_time": 13.965443047881127 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1702.0, + "completions/max_terminated_length": 1172.4, + "completions/mean_length": 1089.1000244140625, + "completions/mean_terminated_length": 908.7400146484375, + "completions/min_length": 716.8, + "completions/min_terminated_length": 716.8, + "entropy": 0.1683452881872654, + "epoch": 0.15664845173041894, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.014243739657104015, + "kl": 0.0004299223937171822, + "learning_rate": 4.4401840490797544e-07, + "loss": -5.882064579054713e-05, + "num_tokens": 2838744.0, + "reward": 1.5133334517478942, + "reward_std": 0.6114970922470093, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.23333333134651185, + "rewards/correctness/std": 0.24494898319244385, + "rewards/multi_component_reward/mean": 0.24666669517755507, + "rewards/multi_component_reward/std": 0.39071274995803834, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.08568799644708633, + "sampling/importance_sampling_ratio/mean": 0.032788149639964105, + "sampling/importance_sampling_ratio/min": 0.0002142171015298561, + "sampling/sampling_logp_difference/max": 0.3399065971374512, + "sampling/sampling_logp_difference/mean": 0.011240929551422595, + "step": 430, + "step_time": 14.933206058479845 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.2666666716337204, + "completions/max_length": 1818.2, + "completions/max_terminated_length": 1232.2, + "completions/mean_length": 1174.3333618164063, + "completions/mean_terminated_length": 910.4933471679688, + "completions/min_length": 678.2, + "completions/min_terminated_length": 678.2, + "entropy": 0.1602409650882085, + "epoch": 0.15846994535519127, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.07029227167367935, + "kl": 0.00047036608011694623, + "learning_rate": 4.4305981595092023e-07, + "loss": -0.004705151170492172, + "num_tokens": 2877094.0, + "reward": 1.4300000429153443, + "reward_std": 0.7005062401294708, + "rewards/boxed_rate/mean": 0.7333333373069764, + "rewards/boxed_rate/std": 0.3977532863616943, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.13000000715255738, + "rewards/multi_component_reward/std": 0.5194472134113312, + "rewards/no_answer_rate/mean": 0.2666666716337204, + "rewards/no_answer_rate/std": 0.3977532863616943, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.24494898319244385, + "sampling/importance_sampling_ratio/max": 0.13022951409220695, + "sampling/importance_sampling_ratio/mean": 0.05298698227852583, + "sampling/importance_sampling_ratio/min": 9.12893192064404e-05, + "sampling/sampling_logp_difference/max": 0.3845913648605347, + "sampling/sampling_logp_difference/mean": 0.01090236771851778, + "step": 435, + "step_time": 15.83292675986886 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1450.6, + "completions/max_terminated_length": 1089.0, + "completions/mean_length": 925.2000366210938, + "completions/mean_terminated_length": 851.3733581542969, + "completions/min_length": 552.8, + "completions/min_terminated_length": 552.8, + "entropy": 0.19520388320088386, + "epoch": 0.16029143897996356, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.11756877601146698, + "kl": 0.0004913257944281213, + "learning_rate": 4.4210122699386497e-07, + "loss": -0.009752783924341202, + "num_tokens": 2907568.0, + "reward": 1.783333420753479, + "reward_std": 0.8172009825706482, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.33333333134651183, + "rewards/correctness/std": 0.35449349880218506, + "rewards/multi_component_reward/mean": 0.41666669249534605, + "rewards/multi_component_reward/std": 0.4753227561712265, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.07866446217522025, + "sampling/importance_sampling_ratio/mean": 0.03093269495293498, + "sampling/importance_sampling_ratio/min": 0.002900865976889124, + "sampling/sampling_logp_difference/max": 0.3607002258300781, + "sampling/sampling_logp_difference/mean": 0.01270847897976637, + "step": 440, + "step_time": 13.279750570654869 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1239.0, + "completions/max_terminated_length": 991.2, + "completions/mean_length": 793.8333618164063, + "completions/mean_terminated_length": 740.3466857910156, + "completions/min_length": 537.4, + "completions/min_terminated_length": 537.4, + "entropy": 0.172708131869634, + "epoch": 0.1621129326047359, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.1338849663734436, + "kl": 0.0005728138513707866, + "learning_rate": 4.411426380368098e-07, + "loss": -0.0031532272696495056, + "num_tokens": 2937617.0, + "reward": 1.6400001287460326, + "reward_std": 0.6522666096687317, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.2666666716337204, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.37333334386348727, + "rewards/multi_component_reward/std": 0.35980162024497986, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07935073487460613, + "sampling/importance_sampling_ratio/mean": 0.024918343313038348, + "sampling/importance_sampling_ratio/min": 0.001050433111959137, + "sampling/sampling_logp_difference/max": 0.37157559394836426, + "sampling/sampling_logp_difference/mean": 0.011105079017579556, + "step": 445, + "step_time": 11.855768189579248 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1251.4, + "completions/max_terminated_length": 829.6, + "completions/mean_length": 769.7333618164063, + "completions/mean_terminated_length": 598.9666748046875, + "completions/min_length": 440.8, + "completions/min_terminated_length": 440.8, + "entropy": 0.16670421635111174, + "epoch": 0.16393442622950818, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.47747763991355896, + "kl": 0.00048284496394141265, + "learning_rate": 4.401840490797546e-07, + "loss": -0.013033266365528106, + "num_tokens": 2963841.0, + "reward": 1.9300000429153443, + "reward_std": 0.8960777290165425, + "rewards/boxed_rate/mean": 0.8666666746139526, + "rewards/boxed_rate/std": 0.20655910968780516, + "rewards/correctness/mean": 0.40000001192092893, + "rewards/correctness/std": 0.41311821937561033, + "rewards/multi_component_reward/mean": 0.42999999821186063, + "rewards/multi_component_reward/std": 0.601305240392685, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.20655910968780516, + "rewards/repetition_rate/mean": 0.10000000298023223, + "rewards/repetition_rate/std": 0.18492921590805053, + "sampling/importance_sampling_ratio/max": 0.14179300479590892, + "sampling/importance_sampling_ratio/mean": 0.06188145540654659, + "sampling/importance_sampling_ratio/min": 0.017718057095797223, + "sampling/sampling_logp_difference/max": 0.3239980101585388, + "sampling/sampling_logp_difference/mean": 0.010319609567523003, + "step": 450, + "step_time": 11.826861386746168 + }, + { + "epoch": 0.16393442622950818, + "eval_clip_ratio/high_max": 0.0, + "eval_clip_ratio/high_mean": 0.0, + "eval_clip_ratio/low_mean": 0.0, + "eval_clip_ratio/low_min": 0.0, + "eval_clip_ratio/region_mean": 0.0, + "eval_completions/clipped_ratio": 0.20666667073965073, + "eval_completions/max_length": 1688.02, + "eval_completions/max_terminated_length": 1185.42, + "eval_completions/mean_length": 1123.9933685302735, + "eval_completions/mean_terminated_length": 846.4790167236329, + "eval_completions/min_length": 676.62, + "eval_completions/min_terminated_length": 594.7, + "eval_entropy": 0.18630945876240732, + "eval_frac_reward_zero_std": 0.16, + "eval_kl": 0.0004587844273191877, + "eval_loss": -0.0019339508144184947, + "eval_num_tokens": 2963841.0, + "eval_reward": 1.5870000886917115, + "eval_reward_std": 0.7317568999528885, + "eval_rewards/boxed_rate/mean": 0.7933333313465118, + "eval_rewards/boxed_rate/std": 0.2689453566074371, + "eval_rewards/correctness/mean": 0.2733333346247673, + "eval_rewards/correctness/std": 0.2873582673072815, + "eval_rewards/multi_component_reward/mean": 0.2603333519771695, + "eval_rewards/multi_component_reward/std": 0.4510752035677433, + "eval_rewards/no_answer_rate/mean": 0.20666667073965073, + "eval_rewards/no_answer_rate/std": 0.2689453566074371, + "eval_rewards/repetition_rate/mean": 0.05333333492279053, + "eval_rewards/repetition_rate/std": 0.09630359530448913, + "eval_runtime": 669.3873, + "eval_samples_per_second": 0.075, + "eval_sampling/importance_sampling_ratio/max": 0.10398212056839838, + "eval_sampling/importance_sampling_ratio/mean": 0.035226596340653483, + "eval_sampling/importance_sampling_ratio/min": 0.0024896363094269146, + "eval_sampling/sampling_logp_difference/max": 0.5326641929149628, + "eval_sampling/sampling_logp_difference/mean": 0.012828467870131135, + "eval_steps_per_second": 0.013, + "step": 450 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1987.6, + "completions/max_terminated_length": 1503.0, + "completions/mean_length": 1150.6666870117188, + "completions/mean_terminated_length": 990.6999938964843, + "completions/min_length": 602.2, + "completions/min_terminated_length": 602.2, + "entropy": 0.2162185865143935, + "epoch": 0.1657559198542805, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01271678227931261, + "kl": 0.0005497580714290962, + "learning_rate": 4.3922546012269933e-07, + "loss": 0.0011618516407907009, + "num_tokens": 3000941.0, + "reward": 1.6966667413711547, + "reward_std": 0.9000118732452392, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.3000000089406967, + "rewards/correctness/std": 0.3914883255958557, + "rewards/multi_component_reward/mean": 0.3300000101327896, + "rewards/multi_component_reward/std": 0.5810832887887954, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.11817909283563495, + "sampling/importance_sampling_ratio/mean": 0.02859312845394015, + "sampling/importance_sampling_ratio/min": 0.00019200471108481247, + "sampling/sampling_logp_difference/max": 0.40274404287338256, + "sampling/sampling_logp_difference/mean": 0.014035884011536836, + "step": 455, + "step_time": 16.644585143402217 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1850.4, + "completions/max_terminated_length": 1004.8, + "completions/mean_length": 914.5666870117187, + "completions/mean_terminated_length": 717.0500061035157, + "completions/min_length": 396.2, + "completions/min_terminated_length": 396.2, + "entropy": 0.18004637969036896, + "epoch": 0.16757741347905283, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.051200754940509796, + "kl": 0.0005335340790528183, + "learning_rate": 4.3826687116564417e-07, + "loss": -0.01714204400777817, + "num_tokens": 3030898.0, + "reward": 1.5366667270660401, + "reward_std": 0.7392153739929199, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.23666667342185974, + "rewards/multi_component_reward/std": 0.4870242774486542, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.1253473086282611, + "sampling/importance_sampling_ratio/mean": 0.06275567463599145, + "sampling/importance_sampling_ratio/min": 0.0048506295410769285, + "sampling/sampling_logp_difference/max": 0.35658023357391355, + "sampling/sampling_logp_difference/mean": 0.012955071125179528, + "step": 460, + "step_time": 15.270139714889229 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1409.8, + "completions/max_terminated_length": 850.4, + "completions/mean_length": 728.4666931152344, + "completions/mean_terminated_length": 629.5000183105469, + "completions/min_length": 469.8, + "completions/min_terminated_length": 469.8, + "entropy": 0.16985683316985767, + "epoch": 0.16939890710382513, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.02098464034497738, + "kl": 0.0005443188643160586, + "learning_rate": 4.3730828220858896e-07, + "loss": -0.007486963272094726, + "num_tokens": 3056190.0, + "reward": 1.9700001001358032, + "reward_std": 0.4863309383392334, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.43333333134651186, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.5366666868329049, + "rewards/multi_component_reward/std": 0.3014017343521118, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.12007005885243416, + "sampling/importance_sampling_ratio/mean": 0.05674390830099583, + "sampling/importance_sampling_ratio/min": 0.0004069998664532642, + "sampling/sampling_logp_difference/max": 0.5247022390365601, + "sampling/sampling_logp_difference/mean": 0.012214668095111847, + "step": 465, + "step_time": 12.395325169526041 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1901.0, + "completions/max_terminated_length": 1594.8, + "completions/mean_length": 1356.2000244140625, + "completions/mean_terminated_length": 1266.056689453125, + "completions/min_length": 993.0, + "completions/min_terminated_length": 993.0, + "entropy": 0.10780975048740706, + "epoch": 0.17122040072859745, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.05852176249027252, + "kl": 0.0004035008855377479, + "learning_rate": 4.363496932515337e-07, + "loss": -0.0037086054682731627, + "num_tokens": 3100044.0, + "reward": 1.6266667366027832, + "reward_std": 0.9011179387569428, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.3823883533477783, + "rewards/multi_component_reward/mean": 0.2933333463966846, + "rewards/multi_component_reward/std": 0.5466254830360413, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.062121391948312524, + "sampling/importance_sampling_ratio/mean": 0.0237305941991508, + "sampling/importance_sampling_ratio/min": 0.004560414611228225, + "sampling/sampling_logp_difference/max": 0.33158082962036134, + "sampling/sampling_logp_difference/mean": 0.007295731455087662, + "step": 470, + "step_time": 16.880542963929475 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1782.4, + "completions/max_terminated_length": 1526.4, + "completions/mean_length": 1103.5000427246093, + "completions/mean_terminated_length": 1016.5600524902344, + "completions/min_length": 741.8, + "completions/min_terminated_length": 741.8, + "entropy": 0.1433272872120142, + "epoch": 0.17304189435336975, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.03871724009513855, + "kl": 0.0003871701846946962, + "learning_rate": 4.353911042944785e-07, + "loss": -0.0024786345660686494, + "num_tokens": 3136959.0, + "reward": 1.1100000619888306, + "reward_std": 0.318433678150177, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.03333333432674408, + "rewards/correctness/std": 0.08164966106414795, + "rewards/multi_component_reward/mean": 0.0766666665673256, + "rewards/multi_component_reward/std": 0.23678402006626129, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.0617107892408967, + "sampling/importance_sampling_ratio/mean": 0.02746242703869939, + "sampling/importance_sampling_ratio/min": 8.897909594907105e-05, + "sampling/sampling_logp_difference/max": 0.3906729698181152, + "sampling/sampling_logp_difference/mean": 0.009637977462261915, + "step": 475, + "step_time": 15.594477642700076 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1867.2, + "completions/max_terminated_length": 1634.6, + "completions/mean_length": 1205.7000244140625, + "completions/mean_terminated_length": 989.493359375, + "completions/min_length": 550.8, + "completions/min_terminated_length": 550.8, + "entropy": 0.16859658484657605, + "epoch": 0.17486338797814208, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.0642157569527626, + "kl": 0.0005781898080992202, + "learning_rate": 4.3443251533742333e-07, + "loss": -0.0028929352760314942, + "num_tokens": 3175776.0, + "reward": 1.3066667556762694, + "reward_std": 0.6569535873830319, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.1066666692495346, + "rewards/multi_component_reward/std": 0.4782795637845993, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.06810643598437309, + "sampling/importance_sampling_ratio/mean": 0.01840525190345943, + "sampling/importance_sampling_ratio/min": 0.0006505777751514535, + "sampling/sampling_logp_difference/max": 0.43449815511703493, + "sampling/sampling_logp_difference/mean": 0.011115654464811086, + "step": 480, + "step_time": 15.972666030749679 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1663.0, + "completions/max_terminated_length": 1303.0, + "completions/mean_length": 1173.3333740234375, + "completions/mean_terminated_length": 989.41669921875, + "completions/min_length": 683.0, + "completions/min_terminated_length": 683.0, + "entropy": 0.14007456911106905, + "epoch": 0.1766848816029144, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.010275622829794884, + "kl": 0.00036965057806810365, + "learning_rate": 4.3347392638036806e-07, + "loss": 0.0001720041735097766, + "num_tokens": 3214522.0, + "reward": 1.9766667842864991, + "reward_std": 0.9272954225540161, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.43333333134651186, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.47666670083999635, + "rewards/multi_component_reward/std": 0.5632504820823669, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.05611779941245913, + "sampling/importance_sampling_ratio/mean": 0.021316760312765838, + "sampling/importance_sampling_ratio/min": 0.0011236143288555823, + "sampling/sampling_logp_difference/max": 0.3524420738220215, + "sampling/sampling_logp_difference/mean": 0.00864975331351161, + "step": 485, + "step_time": 15.02983694653958 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1206.8, + "completions/max_terminated_length": 934.0, + "completions/mean_length": 728.1666748046875, + "completions/mean_terminated_length": 595.9966796875, + "completions/min_length": 426.2, + "completions/min_terminated_length": 426.2, + "entropy": 0.23062201142311095, + "epoch": 0.1785063752276867, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.018157748505473137, + "kl": 0.001033562931115739, + "learning_rate": 4.3251533742331285e-07, + "loss": 0.014268814027309418, + "num_tokens": 3238755.0, + "reward": 1.833333384990692, + "reward_std": 0.6647640764713287, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.3666666686534882, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.43333335071802137, + "rewards/multi_component_reward/std": 0.4189693987369537, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12710050344467164, + "sampling/importance_sampling_ratio/mean": 0.05628317659720779, + "sampling/importance_sampling_ratio/min": 0.015683322838595436, + "sampling/sampling_logp_difference/max": 0.36403354406356814, + "sampling/sampling_logp_difference/mean": 0.015313766896724701, + "step": 490, + "step_time": 11.357261310145258 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1498.2, + "completions/max_terminated_length": 1076.4, + "completions/mean_length": 1013.5000305175781, + "completions/mean_terminated_length": 849.3533508300782, + "completions/min_length": 602.6, + "completions/min_terminated_length": 602.6, + "entropy": 0.14790264442563056, + "epoch": 0.18032786885245902, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.034156352281570435, + "kl": 0.0005029304617589029, + "learning_rate": 4.315567484662577e-07, + "loss": -0.010992055386304855, + "num_tokens": 3274110.0, + "reward": 1.4433334589004516, + "reward_std": 0.6818739175796509, + "rewards/boxed_rate/mean": 0.8333333313465119, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.21000000536441804, + "rewards/multi_component_reward/std": 0.4088459849357605, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.10836826749145985, + "sampling/importance_sampling_ratio/mean": 0.04134368039667606, + "sampling/importance_sampling_ratio/min": 0.0007355699201900378, + "sampling/sampling_logp_difference/max": 0.5285765409469605, + "sampling/sampling_logp_difference/mean": 0.00972840515896678, + "step": 495, + "step_time": 13.717384218610823 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1768.6, + "completions/max_terminated_length": 1172.4, + "completions/mean_length": 1021.0333618164062, + "completions/mean_terminated_length": 829.9766723632813, + "completions/min_length": 574.6, + "completions/min_terminated_length": 574.6, + "entropy": 0.1277496966222922, + "epoch": 0.18214936247723132, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.036998555064201355, + "kl": 0.00041782591021425713, + "learning_rate": 4.3059815950920243e-07, + "loss": -0.014814507961273194, + "num_tokens": 3306649.0, + "reward": 1.9100001335144043, + "reward_std": 0.8856898427009583, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.4333333373069763, + "rewards/correctness/std": 0.3761233925819397, + "rewards/multi_component_reward/mean": 0.4766666904091835, + "rewards/multi_component_reward/std": 0.5212879002094268, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.2542514935135841, + "sampling/importance_sampling_ratio/mean": 0.09570314064621925, + "sampling/importance_sampling_ratio/min": 0.0020768470163739948, + "sampling/sampling_logp_difference/max": 0.34144341945648193, + "sampling/sampling_logp_difference/mean": 0.010077364556491376, + "step": 500, + "step_time": 15.174134342558682 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 1232.6, + "completions/max_terminated_length": 1232.6, + "completions/mean_length": 928.7333618164063, + "completions/mean_terminated_length": 928.7333618164063, + "completions/min_length": 612.8, + "completions/min_terminated_length": 612.8, + "entropy": 0.15919652469456197, + "epoch": 0.18397085610200364, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.03940599039196968, + "kl": 0.0004976143633636335, + "learning_rate": 4.296395705521472e-07, + "loss": 8.051458280533552e-05, + "num_tokens": 3338765.0, + "reward": 1.6133334398269654, + "reward_std": 0.5944257378578186, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.34666666090488435, + "rewards/multi_component_reward/std": 0.3269805133342743, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12865251041948794, + "sampling/importance_sampling_ratio/mean": 0.046758548077195886, + "sampling/importance_sampling_ratio/min": 0.001370607664284762, + "sampling/sampling_logp_difference/max": 0.37642626762390136, + "sampling/sampling_logp_difference/mean": 0.010083456197753549, + "step": 505, + "step_time": 12.115066050365566 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1233.6, + "completions/max_terminated_length": 1011.6, + "completions/mean_length": 741.0666870117187, + "completions/mean_terminated_length": 699.6933471679688, + "completions/min_length": 502.8, + "completions/min_terminated_length": 502.8, + "entropy": 0.14653264259298643, + "epoch": 0.18579234972677597, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.022397559136152267, + "kl": 0.0004972440418593275, + "learning_rate": 4.28680981595092e-07, + "loss": -0.010462169349193574, + "num_tokens": 3363841.0, + "reward": 2.0600001335144045, + "reward_std": 0.9187898755073547, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.4666666626930237, + "rewards/correctness/std": 0.4298781991004944, + "rewards/multi_component_reward/mean": 0.5933333843946457, + "rewards/multi_component_reward/std": 0.49362565875053405, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.16412589382380247, + "sampling/importance_sampling_ratio/mean": 0.0629353643860668, + "sampling/importance_sampling_ratio/min": 0.011790907313888965, + "sampling/sampling_logp_difference/max": 0.3437652587890625, + "sampling/sampling_logp_difference/mean": 0.009886026009917259, + "step": 510, + "step_time": 11.563566098548472 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1427.2, + "completions/max_terminated_length": 666.4, + "completions/mean_length": 705.2000061035156, + "completions/mean_terminated_length": 511.7700134277344, + "completions/min_length": 344.2, + "completions/min_terminated_length": 344.2, + "entropy": 0.26261382003625233, + "epoch": 0.18761384335154827, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.10859289020299911, + "kl": 0.0008684398761639992, + "learning_rate": 4.277223926380368e-07, + "loss": -0.002386706694960594, + "num_tokens": 3386899.0, + "reward": 1.3700001001358033, + "reward_std": 0.7070007205009461, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.20333334654569626, + "rewards/multi_component_reward/std": 0.4274008393287659, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07567991334944964, + "sampling/importance_sampling_ratio/mean": 0.024421159457415343, + "sampling/importance_sampling_ratio/min": 0.001837286539412988, + "sampling/sampling_logp_difference/max": 0.34601569175720215, + "sampling/sampling_logp_difference/mean": 0.017737336456775665, + "step": 515, + "step_time": 12.585359536856412 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1618.2, + "completions/max_terminated_length": 1045.2, + "completions/mean_length": 936.9666870117187, + "completions/mean_terminated_length": 758.3166870117187, + "completions/min_length": 469.8, + "completions/min_terminated_length": 469.8, + "entropy": 0.1936116027335326, + "epoch": 0.1894353369763206, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.0316350944340229, + "kl": 0.0006832293069843824, + "learning_rate": 4.267638036809816e-07, + "loss": -0.008644417673349381, + "num_tokens": 3417600.0, + "reward": 1.5800000786781312, + "reward_std": 0.7550430953502655, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.2666666716337204, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.3133333310484886, + "rewards/multi_component_reward/std": 0.46337632834911346, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.0883142702281475, + "sampling/importance_sampling_ratio/mean": 0.030587387550622226, + "sampling/importance_sampling_ratio/min": 0.0063572791577344466, + "sampling/sampling_logp_difference/max": 0.3835047721862793, + "sampling/sampling_logp_difference/mean": 0.013410749472677707, + "step": 520, + "step_time": 13.994114908203482 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1382.6, + "completions/max_terminated_length": 1288.6, + "completions/mean_length": 903.3000122070313, + "completions/mean_terminated_length": 852.4833374023438, + "completions/min_length": 593.0, + "completions/min_terminated_length": 593.0, + "entropy": 0.18077843909462293, + "epoch": 0.1912568306010929, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.024261396378278732, + "kl": 0.0005541147484715718, + "learning_rate": 4.2580521472392637e-07, + "loss": -0.0031598765403032304, + "num_tokens": 3448035.0, + "reward": 1.8300000786781312, + "reward_std": 0.6672061026096344, + "rewards/boxed_rate/mean": 0.9333333373069763, + "rewards/boxed_rate/std": 0.10327955484390258, + "rewards/correctness/mean": 0.3666666686534882, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.4633333504199982, + "rewards/multi_component_reward/std": 0.378997391462326, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.10327955484390258, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.10047482601366937, + "sampling/importance_sampling_ratio/mean": 0.02871947007952258, + "sampling/importance_sampling_ratio/min": 0.0007279034301063802, + "sampling/sampling_logp_difference/max": 0.41403778791427615, + "sampling/sampling_logp_difference/mean": 0.011369843780994416, + "step": 525, + "step_time": 12.759401306509972 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 1729.4, + "completions/max_terminated_length": 1090.4, + "completions/mean_length": 1117.200048828125, + "completions/mean_terminated_length": 875.8833618164062, + "completions/min_length": 682.2, + "completions/min_terminated_length": 682.2, + "entropy": 0.18201239357391993, + "epoch": 0.1930783242258652, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.00892422441393137, + "kl": 0.000488846098111632, + "learning_rate": 4.2484662576687116e-07, + "loss": -0.004933690279722213, + "num_tokens": 3483861.0, + "reward": 1.3033334016799927, + "reward_std": 0.6930439114570618, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.34822853803634646, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.13666666969656943, + "rewards/multi_component_reward/std": 0.4622148424386978, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.34822853803634646, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.06577782724052668, + "sampling/importance_sampling_ratio/mean": 0.02595797423273325, + "sampling/importance_sampling_ratio/min": 0.0002929158277353877, + "sampling/sampling_logp_difference/max": 0.355899715423584, + "sampling/sampling_logp_difference/mean": 0.012041485961526632, + "step": 530, + "step_time": 15.400034189224243 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1707.4, + "completions/max_terminated_length": 958.8, + "completions/mean_length": 998.1000122070312, + "completions/mean_terminated_length": 766.7933349609375, + "completions/min_length": 525.4, + "completions/min_terminated_length": 525.4, + "entropy": 0.22935964514811832, + "epoch": 0.19489981785063754, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.020105650648474693, + "kl": 0.000557902626072367, + "learning_rate": 4.2388803680981595e-07, + "loss": -0.0007442331407219172, + "num_tokens": 3518562.0, + "reward": 1.2100000977516174, + "reward_std": 0.6208420097827911, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.10000000298023223, + "rewards/correctness/std": 0.24494898319244385, + "rewards/multi_component_reward/mean": 0.11000000834465026, + "rewards/multi_component_reward/std": 0.38656076192855837, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.015947306994348764, + "sampling/importance_sampling_ratio/mean": 0.004562706989236176, + "sampling/importance_sampling_ratio/min": 2.5106548244674746e-05, + "sampling/sampling_logp_difference/max": 0.36506946086883546, + "sampling/sampling_logp_difference/mean": 0.014524615183472634, + "step": 535, + "step_time": 14.785470444336534 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 2011.6, + "completions/max_terminated_length": 1252.0, + "completions/mean_length": 974.2000366210938, + "completions/mean_terminated_length": 799.5733703613281, + "completions/min_length": 499.0, + "completions/min_terminated_length": 499.0, + "entropy": 0.20504479110240936, + "epoch": 0.19672131147540983, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.016418447718024254, + "kl": 0.0007665569457458333, + "learning_rate": 4.229294478527607e-07, + "loss": -0.004444469138979912, + "num_tokens": 3550332.0, + "reward": 1.5333333969116212, + "reward_std": 0.6269734501838684, + "rewards/boxed_rate/mean": 0.8666666507720947, + "rewards/boxed_rate/std": 0.3265986442565918, + "rewards/correctness/mean": 0.23333333134651185, + "rewards/correctness/std": 0.24494898319244385, + "rewards/multi_component_reward/mean": 0.266666679084301, + "rewards/multi_component_reward/std": 0.4252805233001709, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.3265986442565918, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.0710077840834856, + "sampling/importance_sampling_ratio/mean": 0.0242210254073143, + "sampling/importance_sampling_ratio/min": 2.3673296637852498e-05, + "sampling/sampling_logp_difference/max": 0.37494783401489257, + "sampling/sampling_logp_difference/mean": 0.013403672352433204, + "step": 540, + "step_time": 16.360854005627335 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1755.4, + "completions/max_terminated_length": 1206.4, + "completions/mean_length": 827.3000122070313, + "completions/mean_terminated_length": 693.3333374023438, + "completions/min_length": 476.2, + "completions/min_terminated_length": 476.2, + "entropy": 0.20222537256777287, + "epoch": 0.19854280510018216, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.07017825543880463, + "kl": 0.0005556545128153327, + "learning_rate": 4.219708588957055e-07, + "loss": -0.0067356817424297334, + "num_tokens": 3577779.0, + "reward": 1.8100000619888306, + "reward_std": 0.7233119249343872, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.3666666686534882, + "rewards/correctness/std": 0.2882087707519531, + "rewards/multi_component_reward/mean": 0.44333335757255554, + "rewards/multi_component_reward/std": 0.43861203789711, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.07237468343228101, + "sampling/importance_sampling_ratio/mean": 0.02907364722341299, + "sampling/importance_sampling_ratio/min": 0.003791734886874308, + "sampling/sampling_logp_difference/max": 0.3440469026565552, + "sampling/sampling_logp_difference/mean": 0.013698535226285457, + "step": 545, + "step_time": 14.568762712925672 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1740.6, + "completions/max_terminated_length": 1098.0, + "completions/mean_length": 972.5000366210937, + "completions/mean_terminated_length": 813.2100280761719, + "completions/min_length": 548.8, + "completions/min_terminated_length": 548.8, + "entropy": 0.15388319492340088, + "epoch": 0.20036429872495445, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.09412358701229095, + "kl": 0.00047789589831760774, + "learning_rate": 4.210122699386503e-07, + "loss": 0.0004236143082380295, + "num_tokens": 3610650.0, + "reward": 1.7666667222976684, + "reward_std": 0.748459929227829, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.3333333432674408, + "rewards/correctness/std": 0.3098386645317078, + "rewards/multi_component_reward/mean": 0.3666666656732559, + "rewards/multi_component_reward/std": 0.5095339387655258, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.07994540650397539, + "sampling/importance_sampling_ratio/mean": 0.0390994424931705, + "sampling/importance_sampling_ratio/min": 0.0009056368981219976, + "sampling/sampling_logp_difference/max": 0.3753530740737915, + "sampling/sampling_logp_difference/mean": 0.009989727940410375, + "step": 550, + "step_time": 15.042225486785174 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1604.4, + "completions/max_terminated_length": 979.4, + "completions/mean_length": 832.6666870117188, + "completions/mean_terminated_length": 665.1333374023437, + "completions/min_length": 452.0, + "completions/min_terminated_length": 452.0, + "entropy": 0.2048481193681558, + "epoch": 0.20218579234972678, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.08716036379337311, + "kl": 0.000548629568462881, + "learning_rate": 4.2005368098159505e-07, + "loss": -0.005523241683840751, + "num_tokens": 3638024.0, + "reward": 1.6033334016799927, + "reward_std": 0.797308748960495, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.32236858606338503, + "rewards/multi_component_reward/mean": 0.30333334803581236, + "rewards/multi_component_reward/std": 0.5069476455450058, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.07022089585661888, + "sampling/importance_sampling_ratio/mean": 0.022322331066243352, + "sampling/importance_sampling_ratio/min": 0.000656013962759859, + "sampling/sampling_logp_difference/max": 0.3337650179862976, + "sampling/sampling_logp_difference/mean": 0.013166856206953525, + "step": 555, + "step_time": 13.850998834148049 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1198.2, + "completions/max_terminated_length": 1077.2, + "completions/mean_length": 757.1333618164062, + "completions/mean_terminated_length": 690.2466918945313, + "completions/min_length": 448.2, + "completions/min_terminated_length": 448.2, + "entropy": 0.18011298303802808, + "epoch": 0.2040072859744991, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.01096227765083313, + "kl": 0.0006569843031077956, + "learning_rate": 4.1909509202453984e-07, + "loss": -0.007073364406824112, + "num_tokens": 3663726.0, + "reward": 1.9000000715255738, + "reward_std": 0.8417381644248962, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.40000000298023225, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.5000000268220901, + "rewards/multi_component_reward/std": 0.4765937089920044, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.09752159416675568, + "sampling/importance_sampling_ratio/mean": 0.05281178932636976, + "sampling/importance_sampling_ratio/min": 0.003194372950383695, + "sampling/sampling_logp_difference/max": 0.3607391357421875, + "sampling/sampling_logp_difference/mean": 0.011687950044870377, + "step": 560, + "step_time": 11.587789290212095 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1599.4, + "completions/max_terminated_length": 1067.8, + "completions/mean_length": 1085.8000244140626, + "completions/mean_terminated_length": 867.1133544921875, + "completions/min_length": 651.6, + "completions/min_terminated_length": 651.6, + "entropy": 0.12313016926248868, + "epoch": 0.2058287795992714, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.05608590692281723, + "kl": 0.00031730876992999886, + "learning_rate": 4.181365030674847e-07, + "loss": 0.0087483711540699, + "num_tokens": 3699090.0, + "reward": 1.7966667652130126, + "reward_std": 0.8426974892616272, + "rewards/boxed_rate/mean": 0.8, + "rewards/boxed_rate/std": 0.29447373151779177, + "rewards/correctness/mean": 0.36666667461395264, + "rewards/correctness/std": 0.3161036252975464, + "rewards/multi_component_reward/mean": 0.36333334594964983, + "rewards/multi_component_reward/std": 0.5058179676532746, + "rewards/no_answer_rate/mean": 0.20000000298023224, + "rewards/no_answer_rate/std": 0.29447373151779177, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.10868240725831128, + "sampling/importance_sampling_ratio/mean": 0.04146801241877256, + "sampling/importance_sampling_ratio/min": 0.008836518857719966, + "sampling/sampling_logp_difference/max": 0.3564323544502258, + "sampling/sampling_logp_difference/mean": 0.007883247546851635, + "step": 565, + "step_time": 14.454759103059768 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.33333333432674406, + "completions/max_length": 1746.6, + "completions/max_terminated_length": 973.4, + "completions/mean_length": 1263.9000244140625, + "completions/mean_terminated_length": 730.0800231933594, + "completions/min_length": 952.8, + "completions/min_terminated_length": 543.2, + "entropy": 0.14238746762275695, + "epoch": 0.20765027322404372, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.01621309295296669, + "kl": 0.0003668556613168524, + "learning_rate": 4.171779141104294e-07, + "loss": 0.00620177686214447, + "num_tokens": 3740091.0, + "reward": 1.4800001382827759, + "reward_std": 0.5089709401130676, + "rewards/boxed_rate/mean": 0.6666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.2666666626930237, + "rewards/correctness/std": 0.19119417667388916, + "rewards/multi_component_reward/mean": 0.17999999970197678, + "rewards/multi_component_reward/std": 0.30394219756126406, + "rewards/no_answer_rate/mean": 0.33333333432674406, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.09319847710430622, + "sampling/importance_sampling_ratio/mean": 0.04575341045856476, + "sampling/importance_sampling_ratio/min": 0.004630656010958226, + "sampling/sampling_logp_difference/max": 0.3321978569030762, + "sampling/sampling_logp_difference/mean": 0.009725131979212164, + "step": 570, + "step_time": 15.637936590984463 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333432674407, + "completions/max_length": 1493.2, + "completions/max_terminated_length": 1230.4, + "completions/mean_length": 1086.3333618164063, + "completions/mean_terminated_length": 990.8466918945312, + "completions/min_length": 756.6, + "completions/min_terminated_length": 756.6, + "entropy": 0.1577371135354042, + "epoch": 0.20947176684881602, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.017710594460368156, + "kl": 0.0003999709859878446, + "learning_rate": 4.162193251533742e-07, + "loss": -0.0008659601211547851, + "num_tokens": 3778573.0, + "reward": 1.2533334612846374, + "reward_std": 0.6350648760795593, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.19119417667388916, + "rewards/correctness/mean": 0.10000000298023223, + "rewards/correctness/std": 0.24494898319244385, + "rewards/multi_component_reward/mean": 0.12000000327825547, + "rewards/multi_component_reward/std": 0.34919504523277284, + "rewards/no_answer_rate/mean": 0.13333333432674407, + "rewards/no_answer_rate/std": 0.19119417667388916, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.021646471600979565, + "sampling/importance_sampling_ratio/mean": 0.009971876302734018, + "sampling/importance_sampling_ratio/min": 0.0019394792594557403, + "sampling/sampling_logp_difference/max": 0.331281840801239, + "sampling/sampling_logp_difference/mean": 0.01047133533284068, + "step": 575, + "step_time": 13.984978386759758 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1077.0, + "completions/max_terminated_length": 843.4, + "completions/mean_length": 618.633349609375, + "completions/mean_terminated_length": 570.0800170898438, + "completions/min_length": 312.4, + "completions/min_terminated_length": 312.4, + "entropy": 0.18354702678819498, + "epoch": 0.21129326047358835, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.01734095625579357, + "kl": 0.0007040846471985181, + "learning_rate": 4.15260736196319e-07, + "loss": -0.034613341093063354, + "num_tokens": 3799994.0, + "reward": 1.733333420753479, + "reward_std": 0.8469935655593872, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.3000000029802322, + "rewards/correctness/std": 0.404018247127533, + "rewards/multi_component_reward/mean": 0.4000000163912773, + "rewards/multi_component_reward/std": 0.481517493724823, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.2370181828737259, + "sampling/importance_sampling_ratio/mean": 0.07491586618125438, + "sampling/importance_sampling_ratio/min": 0.017679853770823685, + "sampling/sampling_logp_difference/max": 0.2730631709098816, + "sampling/sampling_logp_difference/mean": 0.011196557991206646, + "step": 580, + "step_time": 10.380646949820221 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1423.0, + "completions/max_terminated_length": 1241.8, + "completions/mean_length": 999.9000366210937, + "completions/mean_terminated_length": 926.8766967773438, + "completions/min_length": 742.8, + "completions/min_terminated_length": 742.8, + "entropy": 0.17347905735174815, + "epoch": 0.21311475409836064, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.019545914605259895, + "kl": 0.0004457860350764046, + "learning_rate": 4.143021472392638e-07, + "loss": -0.0001349434838630259, + "num_tokens": 3834629.0, + "reward": 1.4100001215934754, + "reward_std": 0.6349397718906402, + "rewards/boxed_rate/mean": 0.9333333373069763, + "rewards/boxed_rate/std": 0.10327955484390258, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.2433333396911621, + "rewards/multi_component_reward/std": 0.36209596991539, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.10327955484390258, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.04343400467187166, + "sampling/importance_sampling_ratio/mean": 0.013024529488757253, + "sampling/importance_sampling_ratio/min": 0.001408253621775657, + "sampling/sampling_logp_difference/max": 0.5604913473129273, + "sampling/sampling_logp_difference/mean": 0.010900301579385995, + "step": 585, + "step_time": 13.423828819952906 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1522.6, + "completions/max_terminated_length": 1304.6, + "completions/mean_length": 1048.300048828125, + "completions/mean_terminated_length": 953.2133544921875, + "completions/min_length": 677.2, + "completions/min_terminated_length": 677.2, + "entropy": 0.13777906013031802, + "epoch": 0.21493624772313297, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.020055213943123817, + "kl": 0.00046802494471194224, + "learning_rate": 4.1334355828220857e-07, + "loss": 0.0007817570120096206, + "num_tokens": 3868946.0, + "reward": 1.3900001525878907, + "reward_std": 0.7840524554252625, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.1666666716337204, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.22333333939313887, + "rewards/multi_component_reward/std": 0.44443279504776, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.05379525385797024, + "sampling/importance_sampling_ratio/mean": 0.019492477364838123, + "sampling/importance_sampling_ratio/min": 0.0019421720592784908, + "sampling/sampling_logp_difference/max": 0.36746039390563967, + "sampling/sampling_logp_difference/mean": 0.009275554586201907, + "step": 590, + "step_time": 13.933117091469466 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1637.0, + "completions/max_terminated_length": 1173.6, + "completions/mean_length": 936.9000122070313, + "completions/mean_terminated_length": 814.3366760253906, + "completions/min_length": 551.2, + "completions/min_terminated_length": 551.2, + "entropy": 0.1666003334025542, + "epoch": 0.2167577413479053, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.011394626460969448, + "kl": 0.00048624937092730153, + "learning_rate": 4.1238496932515336e-07, + "loss": -6.75798102747649e-05, + "num_tokens": 3899387.0, + "reward": 1.483333444595337, + "reward_std": 0.6897397696971893, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.2500000074505806, + "rewards/multi_component_reward/std": 0.42727351784706114, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.03627656940370798, + "sampling/importance_sampling_ratio/mean": 0.014781518559902906, + "sampling/importance_sampling_ratio/min": 0.0008682889689286911, + "sampling/sampling_logp_difference/max": 0.3049093961715698, + "sampling/sampling_logp_difference/mean": 0.010832818690687418, + "step": 595, + "step_time": 14.285567329451442 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1661.6, + "completions/max_terminated_length": 1382.2, + "completions/mean_length": 1118.2000244140625, + "completions/mean_terminated_length": 938.136669921875, + "completions/min_length": 657.6, + "completions/min_terminated_length": 657.6, + "entropy": 0.17573260565598806, + "epoch": 0.2185792349726776, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.020734934136271477, + "kl": 0.0004598392901243642, + "learning_rate": 4.1142638036809815e-07, + "loss": -0.002467634528875351, + "num_tokens": 3936269.0, + "reward": 1.0933334231376648, + "reward_std": 0.33367283940315245, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.03333333432674408, + "rewards/correctness/std": 0.08164966106414795, + "rewards/multi_component_reward/mean": 0.02666666656732559, + "rewards/multi_component_reward/std": 0.2805217713117599, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.044962253980338575, + "sampling/importance_sampling_ratio/mean": 0.011017974140122533, + "sampling/importance_sampling_ratio/min": 0.0003941454909745216, + "sampling/sampling_logp_difference/max": 0.38812339305877686, + "sampling/sampling_logp_difference/mean": 0.01122636515647173, + "step": 600, + "step_time": 14.878706483915447 + }, + { + "epoch": 0.2185792349726776, + "eval_clip_ratio/high_max": 0.0, + "eval_clip_ratio/high_mean": 0.0, + "eval_clip_ratio/low_mean": 0.0, + "eval_clip_ratio/low_min": 0.0, + "eval_clip_ratio/region_mean": 0.0, + "eval_completions/clipped_ratio": 0.18333333790302275, + "eval_completions/max_length": 1633.0, + "eval_completions/max_terminated_length": 1261.48, + "eval_completions/mean_length": 1074.5033642578126, + "eval_completions/mean_terminated_length": 884.9103503417969, + "eval_completions/min_length": 608.76, + "eval_completions/min_terminated_length": 608.76, + "eval_entropy": 0.1780784809589386, + "eval_frac_reward_zero_std": 0.1, + "eval_kl": 0.000503081139177084, + "eval_loss": -0.002404627623036504, + "eval_num_tokens": 3936269.0, + "eval_reward": 1.5616667437553406, + "eval_reward_std": 0.7698906905949116, + "eval_rewards/boxed_rate/mean": 0.816666665673256, + "eval_rewards/boxed_rate/std": 0.28143630146980286, + "eval_rewards/correctness/mean": 0.260000002682209, + "eval_rewards/correctness/std": 0.3175121474266052, + "eval_rewards/multi_component_reward/mean": 0.26500001683831215, + "eval_rewards/multi_component_reward/std": 0.49374800592660906, + "eval_rewards/no_answer_rate/mean": 0.18333333790302275, + "eval_rewards/no_answer_rate/std": 0.28143630146980286, + "eval_rewards/repetition_rate/mean": 0.036666667759418486, + "eval_rewards/repetition_rate/std": 0.08381265044212341, + "eval_runtime": 648.1074, + "eval_samples_per_second": 0.077, + "eval_sampling/importance_sampling_ratio/max": 0.08480258989613504, + "eval_sampling/importance_sampling_ratio/mean": 0.03125391862238757, + "eval_sampling/importance_sampling_ratio/min": 0.0025257051857090485, + "eval_sampling/sampling_logp_difference/max": 0.5640374529361725, + "eval_sampling/sampling_logp_difference/mean": 0.01284005825407803, + "eval_steps_per_second": 0.014, + "step": 600 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1519.2, + "completions/max_terminated_length": 1256.6, + "completions/mean_length": 946.06669921875, + "completions/mean_terminated_length": 839.1633544921875, + "completions/min_length": 563.2, + "completions/min_terminated_length": 563.2, + "entropy": 0.14678182030717532, + "epoch": 0.2204007285974499, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.023046690970659256, + "kl": 0.0005303417919397664, + "learning_rate": 4.1046779141104294e-07, + "loss": -0.0009497999213635922, + "num_tokens": 3967489.0, + "reward": 1.2500001072883606, + "reward_std": 0.46636478304862977, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.10000000298023223, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.15000001192092896, + "rewards/multi_component_reward/std": 0.28614950776100156, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.15675322916358708, + "sampling/importance_sampling_ratio/mean": 0.057061844225972894, + "sampling/importance_sampling_ratio/min": 0.0010045379561626475, + "sampling/sampling_logp_difference/max": 0.3354309320449829, + "sampling/sampling_logp_difference/mean": 0.010376049391925334, + "step": 605, + "step_time": 13.543810763396323 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1038.6, + "completions/max_terminated_length": 971.6, + "completions/mean_length": 741.6333618164062, + "completions/mean_terminated_length": 710.4466918945312, + "completions/min_length": 523.2, + "completions/min_terminated_length": 523.2, + "entropy": 0.17811203648646673, + "epoch": 0.2222222222222222, + "frac_reward_zero_std": 0.0, + "grad_norm": 1.1156023740768433, + "kl": 0.0007145934030025577, + "learning_rate": 4.095092024539877e-07, + "loss": 0.001612265408039093, + "num_tokens": 3992006.0, + "reward": 1.8733334302902223, + "reward_std": 0.8630595266819, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.3666666656732559, + "rewards/correctness/std": 0.3761233925819397, + "rewards/multi_component_reward/mean": 0.4733333714306355, + "rewards/multi_component_reward/std": 0.4589902624487877, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.12649268209934234, + "sampling/importance_sampling_ratio/mean": 0.06289250953122974, + "sampling/importance_sampling_ratio/min": 0.03086609955687436, + "sampling/sampling_logp_difference/max": 0.3575249195098877, + "sampling/sampling_logp_difference/mean": 0.011591816507279872, + "step": 610, + "step_time": 10.539130839891731 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1739.8, + "completions/max_terminated_length": 1545.4, + "completions/mean_length": 1192.700048828125, + "completions/mean_terminated_length": 1113.4033569335938, + "completions/min_length": 698.0, + "completions/min_terminated_length": 698.0, + "entropy": 0.1430244820813338, + "epoch": 0.22404371584699453, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.008428337052464485, + "kl": 0.00040708660574940346, + "learning_rate": 4.0855061349693246e-07, + "loss": 0.0004372816067188978, + "num_tokens": 4030967.0, + "reward": 2.020000123977661, + "reward_std": 0.8252323269844055, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.4666666746139526, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.5533333420753479, + "rewards/multi_component_reward/std": 0.4648365914821625, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.06130031887441874, + "sampling/importance_sampling_ratio/mean": 0.01900877393782139, + "sampling/importance_sampling_ratio/min": 0.00034280085452338406, + "sampling/sampling_logp_difference/max": 0.37778170108795167, + "sampling/sampling_logp_difference/mean": 0.009492585808038712, + "step": 615, + "step_time": 15.555601417645812 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1618.8, + "completions/max_terminated_length": 932.0, + "completions/mean_length": 796.36669921875, + "completions/mean_terminated_length": 658.6666809082031, + "completions/min_length": 447.4, + "completions/min_terminated_length": 447.4, + "entropy": 0.24623483493924142, + "epoch": 0.22586520947176686, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.031970929354429245, + "kl": 0.0006379926751833409, + "learning_rate": 4.075920245398773e-07, + "loss": -0.0030921820551157, + "num_tokens": 4057612.0, + "reward": 1.3433334350585937, + "reward_std": 0.5004158258438111, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.20655910968780516, + "rewards/multi_component_reward/mean": 0.1766666740179062, + "rewards/multi_component_reward/std": 0.33977961242198945, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.06918634744361044, + "sampling/importance_sampling_ratio/mean": 0.02087077551987022, + "sampling/importance_sampling_ratio/min": 0.0005975837658579486, + "sampling/sampling_logp_difference/max": 0.34571409225463867, + "sampling/sampling_logp_difference/mean": 0.01637477772310376, + "step": 620, + "step_time": 13.71192037537694 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1644.8, + "completions/max_terminated_length": 1458.6, + "completions/mean_length": 1135.6000244140625, + "completions/mean_terminated_length": 976.4267211914063, + "completions/min_length": 573.8, + "completions/min_terminated_length": 573.8, + "entropy": 0.15917381308972836, + "epoch": 0.22768670309653916, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.03386072441935539, + "kl": 0.0005424051094450987, + "learning_rate": 4.066334355828221e-07, + "loss": 0.0008449406363070011, + "num_tokens": 4094980.0, + "reward": 1.4666667938232423, + "reward_std": 0.5768125057220459, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.20655910968780516, + "rewards/multi_component_reward/mean": 0.20000001415610313, + "rewards/multi_component_reward/std": 0.3994357705116272, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.054127467796206474, + "sampling/importance_sampling_ratio/mean": 0.016182091273367404, + "sampling/importance_sampling_ratio/min": 0.002016719186418556, + "sampling/sampling_logp_difference/max": 0.3768587350845337, + "sampling/sampling_logp_difference/mean": 0.009851652942597867, + "step": 625, + "step_time": 14.74142508301884 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1218.6, + "completions/max_terminated_length": 1002.4, + "completions/mean_length": 759.3666870117188, + "completions/mean_terminated_length": 716.2800170898438, + "completions/min_length": 491.2, + "completions/min_terminated_length": 491.2, + "entropy": 0.19249504134058953, + "epoch": 0.22950819672131148, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.040883224457502365, + "kl": 0.0008160913091463347, + "learning_rate": 4.0567484662576683e-07, + "loss": -0.02089705318212509, + "num_tokens": 4121865.0, + "reward": 1.6600001096725463, + "reward_std": 0.8352818369865418, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.2666666716337204, + "rewards/correctness/std": 0.3977532863616943, + "rewards/multi_component_reward/mean": 0.3933333531022072, + "rewards/multi_component_reward/std": 0.4375286281108856, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.12170692235231399, + "sampling/importance_sampling_ratio/mean": 0.031014200672507285, + "sampling/importance_sampling_ratio/min": 0.001588601156254299, + "sampling/sampling_logp_difference/max": 0.3645863771438599, + "sampling/sampling_logp_difference/mean": 0.01216810904443264, + "step": 630, + "step_time": 11.676486384868621 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 1104.8, + "completions/max_terminated_length": 1104.8, + "completions/mean_length": 809.733349609375, + "completions/mean_terminated_length": 809.733349609375, + "completions/min_length": 576.0, + "completions/min_terminated_length": 576.0, + "entropy": 0.17598983322580655, + "epoch": 0.23132969034608378, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.016296500340104103, + "kl": 0.0005092226871056482, + "learning_rate": 4.0471625766871167e-07, + "loss": -0.004767249897122383, + "num_tokens": 4148941.0, + "reward": 1.520000123977661, + "reward_std": 0.6183947920799255, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.3200000077486038, + "rewards/multi_component_reward/std": 0.3239211142063141, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.17697620438411832, + "sampling/importance_sampling_ratio/mean": 0.06787280989810825, + "sampling/importance_sampling_ratio/min": 0.03228051500591391, + "sampling/sampling_logp_difference/max": 0.34911930561065674, + "sampling/sampling_logp_difference/mean": 0.011234280280768871, + "step": 635, + "step_time": 10.972807624004782 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1261.6, + "completions/max_terminated_length": 1218.8, + "completions/mean_length": 815.0000244140625, + "completions/mean_terminated_length": 784.780029296875, + "completions/min_length": 532.2, + "completions/min_terminated_length": 532.2, + "entropy": 0.14789476320147515, + "epoch": 0.2331511839708561, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.08125295490026474, + "kl": 0.0006361469063752641, + "learning_rate": 4.037576687116564e-07, + "loss": -0.008612464368343353, + "num_tokens": 4176637.0, + "reward": 1.7100001096725463, + "reward_std": 0.8422193884849548, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.3000000089406967, + "rewards/correctness/std": 0.3914883255958557, + "rewards/multi_component_reward/mean": 0.41000000983476637, + "rewards/multi_component_reward/std": 0.4542399704456329, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.14553753491491078, + "sampling/importance_sampling_ratio/mean": 0.0640607855282724, + "sampling/importance_sampling_ratio/min": 0.0038686898650384326, + "sampling/sampling_logp_difference/max": 0.3177733182907104, + "sampling/sampling_logp_difference/mean": 0.00996503084897995, + "step": 640, + "step_time": 11.762126618809997 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1707.4, + "completions/max_terminated_length": 1151.8, + "completions/mean_length": 1026.000048828125, + "completions/mean_terminated_length": 833.5666870117187, + "completions/min_length": 601.6, + "completions/min_terminated_length": 601.6, + "entropy": 0.15211746419469516, + "epoch": 0.23497267759562843, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.04184797778725624, + "kl": 0.0004896697394239406, + "learning_rate": 4.027990797546012e-07, + "loss": -0.0011240935884416104, + "num_tokens": 4210123.0, + "reward": 1.7233334302902221, + "reward_std": 0.8806208968162537, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.33333333432674406, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.35666669607162477, + "rewards/multi_component_reward/std": 0.5503794223070144, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.10542454989627004, + "sampling/importance_sampling_ratio/mean": 0.03725480753928423, + "sampling/importance_sampling_ratio/min": 0.0006887080645215871, + "sampling/sampling_logp_difference/max": 0.3569879293441772, + "sampling/sampling_logp_difference/mean": 0.011003307159990072, + "step": 645, + "step_time": 14.911480633541942 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1574.6, + "completions/max_terminated_length": 1237.0, + "completions/mean_length": 920.3000244140625, + "completions/mean_terminated_length": 845.8533569335938, + "completions/min_length": 600.0, + "completions/min_terminated_length": 600.0, + "entropy": 0.2023964905490478, + "epoch": 0.23679417122040072, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.023950839415192604, + "kl": 0.0005954526874120347, + "learning_rate": 4.01840490797546e-07, + "loss": 0.0016510456800460816, + "num_tokens": 4241086.0, + "reward": 1.4766667366027832, + "reward_std": 0.5016432344913483, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2128240704536438, + "rewards/multi_component_reward/mean": 0.24333334639668464, + "rewards/multi_component_reward/std": 0.29857338666915895, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.02175317257642746, + "sampling/importance_sampling_ratio/mean": 0.007160785421729088, + "sampling/importance_sampling_ratio/min": 0.00033338346147502305, + "sampling/sampling_logp_difference/max": 0.387990665435791, + "sampling/sampling_logp_difference/mean": 0.012314715608954429, + "step": 650, + "step_time": 13.970132818445563 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.03333333432674408, + "completions/max_length": 1193.4, + "completions/max_terminated_length": 1015.4, + "completions/mean_length": 716.5000244140625, + "completions/mean_terminated_length": 674.5733520507813, + "completions/min_length": 464.6, + "completions/min_terminated_length": 464.6, + "entropy": 0.16268871029218038, + "epoch": 0.23861566484517305, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.06329948455095291, + "kl": 0.0005586101295193657, + "learning_rate": 4.0088190184049077e-07, + "loss": -0.0020235760137438776, + "num_tokens": 4265005.0, + "reward": 1.5000001192092896, + "reward_std": 0.7927483677864074, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.30000000447034836, + "rewards/multi_component_reward/std": 0.4276039183139801, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.08887637332081795, + "sampling/importance_sampling_ratio/mean": 0.04327146988362074, + "sampling/importance_sampling_ratio/min": 0.01085926350569757, + "sampling/sampling_logp_difference/max": 0.34261035919189453, + "sampling/sampling_logp_difference/mean": 0.010840693116188049, + "step": 655, + "step_time": 11.254522551037372 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1448.4, + "completions/max_terminated_length": 1037.6, + "completions/mean_length": 916.8000366210938, + "completions/mean_terminated_length": 796.9666870117187, + "completions/min_length": 576.2, + "completions/min_terminated_length": 576.2, + "entropy": 0.1381769967575868, + "epoch": 0.24043715846994534, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.018603617325425148, + "kl": 0.0004810667795633587, + "learning_rate": 3.9992331288343556e-07, + "loss": -0.00833008587360382, + "num_tokens": 4294873.0, + "reward": 1.530000126361847, + "reward_std": 0.5924062907695771, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.23333333134651185, + "rewards/correctness/std": 0.24494898319244385, + "rewards/multi_component_reward/mean": 0.29666669070720675, + "rewards/multi_component_reward/std": 0.35217125415802003, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.19538256973028184, + "sampling/importance_sampling_ratio/mean": 0.0801917064934969, + "sampling/importance_sampling_ratio/min": 0.002801872471235356, + "sampling/sampling_logp_difference/max": 0.36407305002212526, + "sampling/sampling_logp_difference/mean": 0.009397336840629577, + "step": 660, + "step_time": 13.188869021646678 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 915.0, + "completions/max_terminated_length": 915.0, + "completions/mean_length": 611.6000183105468, + "completions/mean_terminated_length": 611.6000183105468, + "completions/min_length": 423.0, + "completions/min_terminated_length": 423.0, + "entropy": 0.21465270320574442, + "epoch": 0.24225865209471767, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.021906884387135506, + "kl": 0.0005874778813449666, + "learning_rate": 3.9896472392638035e-07, + "loss": 0.0034822095185518265, + "num_tokens": 4315237.0, + "reward": 1.9400000810623168, + "reward_std": 0.8352818369865418, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.4000000089406967, + "rewards/correctness/std": 0.3977532863616943, + "rewards/multi_component_reward/mean": 0.5399999991059303, + "rewards/multi_component_reward/std": 0.4375286281108856, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.08231882937252522, + "sampling/importance_sampling_ratio/mean": 0.03466434141155332, + "sampling/importance_sampling_ratio/min": 0.0024839662786689588, + "sampling/sampling_logp_difference/max": 0.35751450061798096, + "sampling/sampling_logp_difference/mean": 0.013147631753236055, + "step": 665, + "step_time": 9.326043193787337 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1493.4, + "completions/max_terminated_length": 1139.0, + "completions/mean_length": 914.633349609375, + "completions/mean_terminated_length": 779.0366821289062, + "completions/min_length": 560.8, + "completions/min_terminated_length": 560.8, + "entropy": 0.15737735082705814, + "epoch": 0.24408014571949, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.01330585964024067, + "kl": 0.0005065599815376724, + "learning_rate": 3.9800613496932514e-07, + "loss": -0.006443501263856888, + "num_tokens": 4345802.0, + "reward": 1.32000013589859, + "reward_std": 0.6418773233890533, + "rewards/boxed_rate/mean": 0.9, + "rewards/boxed_rate/std": 0.18492921590805053, + "rewards/correctness/mean": 0.13333333730697633, + "rewards/correctness/std": 0.2665788769721985, + "rewards/multi_component_reward/mean": 0.18666667491197586, + "rewards/multi_component_reward/std": 0.3788073122501373, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.18492921590805053, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.10360531127080322, + "sampling/importance_sampling_ratio/mean": 0.05636316137388349, + "sampling/importance_sampling_ratio/min": 0.022174429874797168, + "sampling/sampling_logp_difference/max": 0.37078907489776614, + "sampling/sampling_logp_difference/mean": 0.010791171807795763, + "step": 670, + "step_time": 13.329693194106222 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1439.8, + "completions/max_terminated_length": 1202.4, + "completions/mean_length": 803.2666870117188, + "completions/mean_terminated_length": 723.4000122070313, + "completions/min_length": 409.0, + "completions/min_terminated_length": 409.0, + "entropy": 0.17263933196663855, + "epoch": 0.2459016393442623, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.1460752934217453, + "kl": 0.0005690245316751923, + "learning_rate": 3.970475460122699e-07, + "loss": -0.012128566205501557, + "num_tokens": 4373710.0, + "reward": 1.6166667461395263, + "reward_std": 0.8088707447052002, + "rewards/boxed_rate/mean": 0.9666666626930237, + "rewards/boxed_rate/std": 0.08164966106414795, + "rewards/correctness/mean": 0.23333334028720856, + "rewards/correctness/std": 0.3914883255958557, + "rewards/multi_component_reward/mean": 0.31666667610406873, + "rewards/multi_component_reward/std": 0.4805600345134735, + "rewards/no_answer_rate/mean": 0.03333333432674408, + "rewards/no_answer_rate/std": 0.08164966106414795, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.10961102806031704, + "sampling/importance_sampling_ratio/mean": 0.043116622418165204, + "sampling/importance_sampling_ratio/min": 0.001660405399086784, + "sampling/sampling_logp_difference/max": 0.3272708892822266, + "sampling/sampling_logp_difference/mean": 0.011050505377352238, + "step": 675, + "step_time": 12.905224220454693 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333333432674408, + "completions/max_length": 1583.4, + "completions/max_terminated_length": 1101.4, + "completions/mean_length": 1081.7000366210937, + "completions/mean_terminated_length": 832.26669921875, + "completions/min_length": 648.0, + "completions/min_terminated_length": 648.0, + "entropy": 0.17594740043083826, + "epoch": 0.24772313296903462, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.07136747241020203, + "kl": 0.0005246107791511653, + "learning_rate": 3.960889570552147e-07, + "loss": 0.003443557769060135, + "num_tokens": 4409875.0, + "reward": 1.4266667485237121, + "reward_std": 0.781731303036213, + "rewards/boxed_rate/mean": 0.7666666626930236, + "rewards/boxed_rate/std": 0.30073869228363037, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.16000001206994058, + "rewards/multi_component_reward/std": 0.509483140707016, + "rewards/no_answer_rate/mean": 0.23333333432674408, + "rewards/no_answer_rate/std": 0.30073869228363037, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.07207991853356362, + "sampling/importance_sampling_ratio/mean": 0.018585072737187146, + "sampling/importance_sampling_ratio/min": 0.00033181315377639464, + "sampling/sampling_logp_difference/max": 0.35654211044311523, + "sampling/sampling_logp_difference/mean": 0.011251153517514467, + "step": 680, + "step_time": 14.236122718639672 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.23333334028720856, + "completions/max_length": 1406.2, + "completions/max_terminated_length": 905.2, + "completions/mean_length": 937.7333770751953, + "completions/mean_terminated_length": 666.7800079345703, + "completions/min_length": 499.4, + "completions/min_terminated_length": 499.4, + "entropy": 0.15866130826373895, + "epoch": 0.2495446265938069, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.019822267815470695, + "kl": 0.0025144809878838714, + "learning_rate": 3.9513036809815945e-07, + "loss": 0.0014299599453806878, + "num_tokens": 4441367.0, + "reward": 1.5433334589004517, + "reward_std": 0.8197273433208465, + "rewards/boxed_rate/mean": 0.7666666686534882, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.26666666865348815, + "rewards/correctness/std": 0.32236858606338503, + "rewards/multi_component_reward/mean": 0.2433333471417427, + "rewards/multi_component_reward/std": 0.5061476826667786, + "rewards/no_answer_rate/mean": 0.23333334028720856, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.19941046759486197, + "sampling/importance_sampling_ratio/mean": 0.10512657705694436, + "sampling/importance_sampling_ratio/min": 0.052634853626081814, + "sampling/sampling_logp_difference/max": 0.39468674659729003, + "sampling/sampling_logp_difference/mean": 0.010894744656980038, + "step": 685, + "step_time": 13.195306334272027 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1319.4, + "completions/max_terminated_length": 1112.6, + "completions/mean_length": 827.1333801269532, + "completions/mean_terminated_length": 760.5666931152343, + "completions/min_length": 536.4, + "completions/min_terminated_length": 536.4, + "entropy": 0.18860391999284426, + "epoch": 0.25136612021857924, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.028540268540382385, + "kl": 0.0006184172428523501, + "learning_rate": 3.941717791411043e-07, + "loss": -0.008557839691638947, + "num_tokens": 4468635.0, + "reward": 2.013333463668823, + "reward_std": 0.8281045794487, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.4333333343267441, + "rewards/correctness/std": 0.3761233925819397, + "rewards/multi_component_reward/mean": 0.513333360850811, + "rewards/multi_component_reward/std": 0.4702867269515991, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.06727873869240283, + "sampling/importance_sampling_ratio/mean": 0.03433487480506301, + "sampling/importance_sampling_ratio/min": 0.00048347263269847286, + "sampling/sampling_logp_difference/max": 0.42853312492370604, + "sampling/sampling_logp_difference/mean": 0.012394568137824535, + "step": 690, + "step_time": 12.268561779148877 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1604.2, + "completions/max_terminated_length": 1084.2, + "completions/mean_length": 907.400048828125, + "completions/mean_terminated_length": 778.053369140625, + "completions/min_length": 563.2, + "completions/min_terminated_length": 563.2, + "entropy": 0.14893935720125834, + "epoch": 0.25318761384335153, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.018119433894753456, + "kl": 0.00045477717115621394, + "learning_rate": 3.932131901840491e-07, + "loss": -0.002763097546994686, + "num_tokens": 4499319.0, + "reward": 1.3900001049041748, + "reward_std": 0.6540532588958741, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.16666666865348817, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.2233333483338356, + "rewards/multi_component_reward/std": 0.3906372755765915, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.09045649766921997, + "sampling/importance_sampling_ratio/mean": 0.034694204293191436, + "sampling/importance_sampling_ratio/min": 0.0024610218745878347, + "sampling/sampling_logp_difference/max": 0.3914134740829468, + "sampling/sampling_logp_difference/mean": 0.010633220244199038, + "step": 695, + "step_time": 13.952417024224996 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1497.8, + "completions/max_terminated_length": 771.4, + "completions/mean_length": 722.333349609375, + "completions/mean_terminated_length": 584.6333435058593, + "completions/min_length": 428.8, + "completions/min_terminated_length": 428.8, + "entropy": 0.20499998504916828, + "epoch": 0.2550091074681239, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.06937753409147263, + "kl": 0.0006178093899507076, + "learning_rate": 3.922546012269938e-07, + "loss": -0.01590590924024582, + "num_tokens": 4523047.0, + "reward": 1.4600000619888305, + "reward_std": 0.7687346935272217, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.20000000596046447, + "rewards/correctness/std": 0.3098386645317078, + "rewards/multi_component_reward/mean": 0.2600000202655792, + "rewards/multi_component_reward/std": 0.4624049305915833, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.16114517226815223, + "sampling/importance_sampling_ratio/mean": 0.06391621977090836, + "sampling/importance_sampling_ratio/min": 0.010175443676988779, + "sampling/sampling_logp_difference/max": 0.3058194637298584, + "sampling/sampling_logp_difference/mean": 0.013935025222599507, + "step": 700, + "step_time": 13.013086147420108 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.16666666865348817, + "completions/max_length": 1429.0, + "completions/max_terminated_length": 942.0, + "completions/mean_length": 947.1667053222657, + "completions/mean_terminated_length": 811.6733581542969, + "completions/min_length": 627.2, + "completions/min_terminated_length": 627.2, + "entropy": 0.17254207134246827, + "epoch": 0.2568306010928962, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.012989591807126999, + "kl": 0.0013695822664885782, + "learning_rate": 3.9129601226993866e-07, + "loss": 0.02462577521800995, + "num_tokens": 4554762.0, + "reward": 2.000000071525574, + "reward_std": 1.0529863953590393, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.4666666626930237, + "rewards/correctness/std": 0.4298781991004944, + "rewards/multi_component_reward/mean": 0.5000000357627868, + "rewards/multi_component_reward/std": 0.5574271440505981, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.1705483404919505, + "sampling/importance_sampling_ratio/mean": 0.062803098349832, + "sampling/importance_sampling_ratio/min": 0.011600531370969901, + "sampling/sampling_logp_difference/max": 0.36405808925628663, + "sampling/sampling_logp_difference/mean": 0.01186208426952362, + "step": 705, + "step_time": 13.669213125482202 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1826.4, + "completions/max_terminated_length": 1155.4, + "completions/mean_length": 1059.6667114257812, + "completions/mean_terminated_length": 841.7533569335938, + "completions/min_length": 515.2, + "completions/min_terminated_length": 515.2, + "entropy": 0.23051666493217152, + "epoch": 0.2586520947176685, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.04497559741139412, + "kl": 0.000664207028845946, + "learning_rate": 3.9033742331288344e-07, + "loss": -0.00020618487615138293, + "num_tokens": 4589378.0, + "reward": 1.8600000977516173, + "reward_std": 0.9444810330867768, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.40000000298023225, + "rewards/correctness/std": 0.36985843181610106, + "rewards/multi_component_reward/mean": 0.4266666829586029, + "rewards/multi_component_reward/std": 0.552643096446991, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.04061034563928843, + "sampling/importance_sampling_ratio/mean": 0.008759467327035964, + "sampling/importance_sampling_ratio/min": 0.0002094253270666409, + "sampling/sampling_logp_difference/max": 0.3215909957885742, + "sampling/sampling_logp_difference/mean": 0.014831000193953513, + "step": 710, + "step_time": 15.66096483822912 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.06666666865348816, + "completions/max_length": 1843.8, + "completions/max_terminated_length": 1615.2, + "completions/mean_length": 1085.3667114257812, + "completions/mean_terminated_length": 1008.9400390625, + "completions/min_length": 637.2, + "completions/min_terminated_length": 637.2, + "entropy": 0.19345977554718655, + "epoch": 0.2604735883424408, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.04418517276644707, + "kl": 0.0008112916897516697, + "learning_rate": 3.893788343558282e-07, + "loss": -0.0014110441319644452, + "num_tokens": 4626127.0, + "reward": 1.5000000953674317, + "reward_std": 0.7173020958900451, + "rewards/boxed_rate/mean": 0.9333333253860474, + "rewards/boxed_rate/std": 0.1632993221282959, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.29447373151779177, + "rewards/multi_component_reward/mean": 0.266666679084301, + "rewards/multi_component_reward/std": 0.3763957768678665, + "rewards/no_answer_rate/mean": 0.06666666865348816, + "rewards/no_answer_rate/std": 0.1632993221282959, + "rewards/repetition_rate/mean": 0.03333333432674408, + "rewards/repetition_rate/std": 0.08164966106414795, + "sampling/importance_sampling_ratio/max": 0.04309340715408325, + "sampling/importance_sampling_ratio/mean": 0.012624496826902032, + "sampling/importance_sampling_ratio/min": 0.00014569541402083248, + "sampling/sampling_logp_difference/max": 0.3676969051361084, + "sampling/sampling_logp_difference/mean": 0.01275388365611434, + "step": 715, + "step_time": 15.848505285196007 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.1666666716337204, + "completions/max_length": 1794.8, + "completions/max_terminated_length": 1234.6, + "completions/mean_length": 973.4000183105469, + "completions/mean_terminated_length": 763.0933410644532, + "completions/min_length": 480.8, + "completions/min_terminated_length": 480.8, + "entropy": 0.19203649039069812, + "epoch": 0.26229508196721313, + "frac_reward_zero_std": 0.2, + "grad_norm": 0.01788559928536415, + "kl": 0.0007585312666681906, + "learning_rate": 3.8842024539877297e-07, + "loss": -0.0016064226627349853, + "num_tokens": 4659067.0, + "reward": 1.3966667771339416, + "reward_std": 0.3670642614364624, + "rewards/boxed_rate/mean": 0.8333333373069763, + "rewards/boxed_rate/std": 0.2882087707519531, + "rewards/correctness/mean": 0.1666666626930237, + "rewards/correctness/std": 0.08164966106414795, + "rewards/multi_component_reward/mean": 0.16333335414528846, + "rewards/multi_component_reward/std": 0.28173157572746277, + "rewards/no_answer_rate/mean": 0.1666666716337204, + "rewards/no_answer_rate/std": 0.2882087707519531, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.1632993221282959, + "sampling/importance_sampling_ratio/max": 0.2071637876331806, + "sampling/importance_sampling_ratio/mean": 0.051915486436337234, + "sampling/importance_sampling_ratio/min": 3.873565276598517e-06, + "sampling/sampling_logp_difference/max": 0.34898459911346436, + "sampling/sampling_logp_difference/mean": 0.013108029030263424, + "step": 720, + "step_time": 15.256685120798647 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000596046447, + "completions/max_length": 1793.6, + "completions/max_terminated_length": 1117.6, + "completions/mean_length": 1055.9667114257813, + "completions/mean_terminated_length": 758.4666870117187, + "completions/min_length": 497.0, + "completions/min_terminated_length": 497.0, + "entropy": 0.20671888664364815, + "epoch": 0.2641165755919854, + "frac_reward_zero_std": 0.4, + "grad_norm": 0.023016415536403656, + "kl": 0.0006531716018798762, + "learning_rate": 3.874616564417178e-07, + "loss": 0.0004923004657030105, + "num_tokens": 4692996.0, + "reward": 1.6100001096725465, + "reward_std": 0.5153546214103699, + "rewards/boxed_rate/mean": 0.7999999940395355, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.3000000029802322, + "rewards/correctness/std": 0.18492921590805053, + "rewards/multi_component_reward/mean": 0.310000017285347, + "rewards/multi_component_reward/std": 0.3351393073797226, + "rewards/no_answer_rate/mean": 0.20000000596046447, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.050453129876405, + "sampling/importance_sampling_ratio/mean": 0.014345021429471671, + "sampling/importance_sampling_ratio/min": 0.0007461899496814892, + "sampling/sampling_logp_difference/max": 0.4133040189743042, + "sampling/sampling_logp_difference/mean": 0.012775494437664747, + "step": 725, + "step_time": 15.257122083753348 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.20000000298023224, + "completions/max_length": 1528.6, + "completions/max_terminated_length": 1156.6, + "completions/mean_length": 1054.86669921875, + "completions/mean_terminated_length": 859.533349609375, + "completions/min_length": 625.0, + "completions/min_terminated_length": 625.0, + "entropy": 0.21140047113100688, + "epoch": 0.2659380692167577, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.018997710198163986, + "kl": 0.0006346999696688726, + "learning_rate": 3.8650306748466255e-07, + "loss": 0.0016422737389802933, + "num_tokens": 4727378.0, + "reward": 1.5333333969116212, + "reward_std": 0.7559072077274323, + "rewards/boxed_rate/mean": 0.8333333253860473, + "rewards/boxed_rate/std": 0.2728438377380371, + "rewards/correctness/mean": 0.23333333432674408, + "rewards/correctness/std": 0.30073869228363037, + "rewards/multi_component_reward/mean": 0.23333334922790527, + "rewards/multi_component_reward/std": 0.4953298598527908, + "rewards/no_answer_rate/mean": 0.16666666865348817, + "rewards/no_answer_rate/std": 0.2728438377380371, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.08731604730710388, + "sampling/importance_sampling_ratio/mean": 0.020629775966517626, + "sampling/importance_sampling_ratio/min": 0.0005315004007400171, + "sampling/sampling_logp_difference/max": 0.3350215911865234, + "sampling/sampling_logp_difference/mean": 0.013690494932234287, + "step": 730, + "step_time": 13.985034779645503 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.13333333730697633, + "completions/max_length": 1815.2, + "completions/max_terminated_length": 1340.4, + "completions/mean_length": 1031.46669921875, + "completions/mean_terminated_length": 883.0133483886718, + "completions/min_length": 518.6, + "completions/min_terminated_length": 518.6, + "entropy": 0.18139945759127538, + "epoch": 0.2677595628415301, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.021211547777056694, + "kl": 0.000629005294952852, + "learning_rate": 3.8554447852760733e-07, + "loss": 0.0014220052398741245, + "num_tokens": 4761916.0, + "reward": 1.4866667628288268, + "reward_std": 0.8185165703296662, + "rewards/boxed_rate/mean": 0.8666666626930237, + "rewards/boxed_rate/std": 0.2665788769721985, + "rewards/correctness/mean": 0.20000000298023224, + "rewards/correctness/std": 0.35449349880218506, + "rewards/multi_component_reward/mean": 0.2200000137090683, + "rewards/multi_component_reward/std": 0.5127273321151733, + "rewards/no_answer_rate/mean": 0.13333333730697633, + "rewards/no_answer_rate/std": 0.2665788769721985, + "rewards/repetition_rate/mean": 0.06666666865348816, + "rewards/repetition_rate/std": 0.10327955484390258, + "sampling/importance_sampling_ratio/max": 0.06554706878960133, + "sampling/importance_sampling_ratio/mean": 0.02508344785310328, + "sampling/importance_sampling_ratio/min": 0.0035423538342001966, + "sampling/sampling_logp_difference/max": 0.31146485805511476, + "sampling/sampling_logp_difference/mean": 0.0113022543489933, + "step": 735, + "step_time": 15.48011973593384 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.0, + "completions/max_length": 883.8, + "completions/max_terminated_length": 883.8, + "completions/mean_length": 542.2000122070312, + "completions/mean_terminated_length": 542.2000122070312, + "completions/min_length": 352.4, + "completions/min_terminated_length": 352.4, + "entropy": 0.19473386829098066, + "epoch": 0.26958105646630237, + "frac_reward_zero_std": 0.4, + "grad_norm": 2.118746042251587, + "kl": 0.0021321156071887042, + "learning_rate": 3.845858895705521e-07, + "loss": -0.0064903564751148226, + "num_tokens": 4781002.0, + "reward": 1.7300001382827759, + "reward_std": 0.5729720234870911, + "rewards/boxed_rate/mean": 1.0, + "rewards/boxed_rate/std": 0.0, + "rewards/correctness/mean": 0.29999999701976776, + "rewards/correctness/std": 0.2728438377380371, + "rewards/multi_component_reward/mean": 0.430000028014183, + "rewards/multi_component_reward/std": 0.3001282274723053, + "rewards/no_answer_rate/mean": 0.0, + "rewards/no_answer_rate/std": 0.0, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.28090705554932355, + "sampling/importance_sampling_ratio/mean": 0.15755705069750547, + "sampling/importance_sampling_ratio/min": 0.0855626948821282, + "sampling/sampling_logp_difference/max": 0.3150986671447754, + "sampling/sampling_logp_difference/mean": 0.01262940764427185, + "step": 740, + "step_time": 9.168580058775841 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.10000000298023223, + "completions/max_length": 1972.8, + "completions/max_terminated_length": 1486.4, + "completions/mean_length": 1172.000048828125, + "completions/mean_terminated_length": 1050.6600280761718, + "completions/min_length": 720.8, + "completions/min_terminated_length": 720.8, + "entropy": 0.11767480584482352, + "epoch": 0.27140255009107467, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.08145014941692352, + "kl": 0.0006028921416145749, + "learning_rate": 3.836273006134969e-07, + "loss": 0.0025216279551386832, + "num_tokens": 4818496.0, + "reward": 1.6700001001358031, + "reward_std": 0.8164093613624572, + "rewards/boxed_rate/mean": 0.899999988079071, + "rewards/boxed_rate/std": 0.24494898319244385, + "rewards/correctness/mean": 0.3, + "rewards/correctness/std": 0.34822853803634646, + "rewards/multi_component_reward/mean": 0.37000002562999723, + "rewards/multi_component_reward/std": 0.47640362083911897, + "rewards/no_answer_rate/mean": 0.10000000298023223, + "rewards/no_answer_rate/std": 0.24494898319244385, + "rewards/repetition_rate/mean": 0.0, + "rewards/repetition_rate/std": 0.0, + "sampling/importance_sampling_ratio/max": 0.14499615281820297, + "sampling/importance_sampling_ratio/mean": 0.04840177483856678, + "sampling/importance_sampling_ratio/min": 0.0013758533679049771, + "sampling/sampling_logp_difference/max": 0.32971014976501467, + "sampling/sampling_logp_difference/mean": 0.008220831863582134, + "step": 745, + "step_time": 16.791488209925593 + }, + { + "clip_ratio/high_max": 0.0, + "clip_ratio/high_mean": 0.0, + "clip_ratio/low_mean": 0.0, + "clip_ratio/low_min": 0.0, + "clip_ratio/region_mean": 0.0, + "completions/clipped_ratio": 0.36666667759418486, + "completions/max_length": 1719.0, + "completions/max_terminated_length": 1232.4, + "completions/mean_length": 1217.600048828125, + "completions/mean_terminated_length": 822.3233520507813, + "completions/min_length": 564.6, + "completions/min_terminated_length": 564.6, + "entropy": 0.19243225703636804, + "epoch": 0.273224043715847, + "frac_reward_zero_std": 0.0, + "grad_norm": 0.03210171312093735, + "kl": 0.0007616788071269791, + "learning_rate": 3.826687116564417e-07, + "loss": -0.005124400556087494, + "num_tokens": 4864732.0, + "reward": 1.7300000667572022, + "reward_std": 0.9617222905158996, + "rewards/boxed_rate/mean": 0.7666666626930236, + "rewards/boxed_rate/std": 0.3761233925819397, + "rewards/correctness/mean": 0.33333334028720857, + "rewards/correctness/std": 0.3977532863616943, + "rewards/multi_component_reward/mean": 0.29666668772697447, + "rewards/multi_component_reward/std": 0.6426073163747787, + "rewards/no_answer_rate/mean": 0.2333333373069763, + "rewards/no_answer_rate/std": 0.3761233925819397, + "rewards/repetition_rate/mean": 0.1, + "rewards/repetition_rate/std": 0.10954451560974121, + "sampling/importance_sampling_ratio/max": 0.11619223132729531, + "sampling/importance_sampling_ratio/mean": 0.03387247350765392, + "sampling/importance_sampling_ratio/min": 0.0005780367053725346, + "sampling/sampling_logp_difference/max": 0.3284996747970581, + "sampling/sampling_logp_difference/mean": 0.012111451011151075, + "step": 750, + "step_time": 15.568404103443026 + }, + { + "epoch": 0.273224043715847, + "eval_clip_ratio/high_max": 0.0, + "eval_clip_ratio/high_mean": 0.0, + "eval_clip_ratio/low_mean": 0.0, + "eval_clip_ratio/low_min": 0.0, + "eval_clip_ratio/region_mean": 0.0, + "eval_completions/clipped_ratio": 0.17000000417232514, + "eval_completions/max_length": 1756.9, + "eval_completions/max_terminated_length": 1208.56, + "eval_completions/mean_length": 1074.366694946289, + "eval_completions/mean_terminated_length": 874.3350158691406, + "eval_completions/min_length": 579.92, + "eval_completions/min_terminated_length": 579.92, + "eval_entropy": 0.18485607638955115, + "eval_frac_reward_zero_std": 0.1, + "eval_kl": 0.0007376758259488269, + "eval_loss": -0.0014031080063432455, + "eval_num_tokens": 4864732.0, + "eval_reward": 1.6413334274291993, + "eval_reward_std": 0.7854073830693961, + "eval_rewards/boxed_rate/mean": 0.8366666615009308, + "eval_rewards/boxed_rate/std": 0.27878632068634035, + "eval_rewards/correctness/mean": 0.2933333370089531, + "eval_rewards/correctness/std": 0.32616410493850706, + "eval_rewards/multi_component_reward/mean": 0.3146666841953993, + "eval_rewards/multi_component_reward/std": 0.4994544792175293, + "eval_rewards/no_answer_rate/mean": 0.16333333730697633, + "eval_rewards/no_answer_rate/std": 0.27878632068634035, + "eval_rewards/repetition_rate/mean": 0.03333333432674408, + "eval_rewards/repetition_rate/std": 0.06964570760726929, + "eval_runtime": 681.9794, + "eval_samples_per_second": 0.073, + "eval_sampling/importance_sampling_ratio/max": 0.09388977923779748, + "eval_sampling/importance_sampling_ratio/mean": 0.029817550851148553, + "eval_sampling/importance_sampling_ratio/min": 0.003447885300730625, + "eval_sampling/sampling_logp_difference/max": 0.7785206460952758, + "eval_sampling/sampling_logp_difference/mean": 0.014042644733563066, + "eval_steps_per_second": 0.013, + "step": 750 + } + ], + "logging_steps": 5, + "max_steps": 2745, + "num_input_tokens_seen": 4864732, + "num_train_epochs": 1, + "save_steps": 150, + "stateful_callbacks": { + "TrainerControl": { + "args": { + "should_epoch_stop": false, + "should_evaluate": false, + "should_log": false, + "should_save": true, + "should_training_stop": false + }, + "attributes": {} + } + }, + "total_flos": 0.0, + "train_batch_size": 1, + "trial_name": null, + "trial_params": null +} diff --git a/training_args.bin b/training_args.bin new file mode 100644 index 0000000000000000000000000000000000000000..133cb6661b0124bb9db0d7d208d748fd736ea689 --- /dev/null +++ b/training_args.bin @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6696e1fef8bde5e22de2f5ae18c284f2b2d35a345157db5151571518385fb38b +size 5713