Instructions to use zerowsw/coderm-8b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use zerowsw/coderm-8b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="zerowsw/coderm-8b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("zerowsw/coderm-8b")
model = AutoModelForCausalLM.from_pretrained("zerowsw/coderm-8b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use zerowsw/coderm-8b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "zerowsw/coderm-8b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerowsw/coderm-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/zerowsw/coderm-8b

SGLang

How to use zerowsw/coderm-8b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "zerowsw/coderm-8b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerowsw/coderm-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "zerowsw/coderm-8b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "zerowsw/coderm-8b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use zerowsw/coderm-8b with Docker Model Runner:
```
docker model run hf.co/zerowsw/coderm-8b
```

zerowsw commited on Mar 5

Commit

1c7c369

verified ·

1 Parent(s): c031906

Model save

Browse files

Files changed (7) hide show

README.md +1 -3
adapter_config.json +2 -2
adapter_model.safetensors +1 -1
all_results.json +5 -5
train_results.json +5 -5
trainer_state.json +488 -952
training_args.bin +1 -1

README.md CHANGED Viewed

@@ -1,11 +1,9 @@
 ---
 base_model: KAKA22/CodeRM-8B
-datasets: zzzzit/taco2
 library_name: transformers
 model_name: coderm-8b
 tags:
 - generated_from_trainer
-- open-r1
 - trl
 - grpo
 licence: license
@@ -13,7 +11,7 @@ licence: license
 # Model Card for coderm-8b
-This model is a fine-tuned version of [KAKA22/CodeRM-8B](https://huggingface.co/KAKA22/CodeRM-8B) on the [zzzzit/taco2](https://huggingface.co/datasets/zzzzit/taco2) dataset.
 It has been trained using [TRL](https://github.com/huggingface/trl).
 ## Quick start

 ---
 base_model: KAKA22/CodeRM-8B
 library_name: transformers
 model_name: coderm-8b
 tags:
 - generated_from_trainer
 - trl
 - grpo
 licence: license
 # Model Card for coderm-8b
+This model is a fine-tuned version of [KAKA22/CodeRM-8B](https://huggingface.co/KAKA22/CodeRM-8B).
 It has been trained using [TRL](https://github.com/huggingface/trl).
 ## Quick start

adapter_config.json CHANGED Viewed

@@ -29,12 +29,12 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
     "v_proj",
     "up_proj",
     "o_proj",
-    "gate_proj",
     "k_proj",
-    "down_proj",
     "q_proj"
   ],
   "target_parameters": null,

   "rank_pattern": {},
   "revision": null,
   "target_modules": [
+    "down_proj",
     "v_proj",
+    "gate_proj",
     "up_proj",
     "o_proj",
     "k_proj",
     "q_proj"
   ],
   "target_parameters": null,

adapter_model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:984bbddd84d801c5546de1a533914fdc16fb1468505b572ab77aaf9edcdc5117
 size 83946192

 version https://git-lfs.github.com/spec/v1
+oid sha256:db711b7e077aa8e969fc5768e16e2c68109039c2a9f73dadff663f7dcee7905e
 size 83946192

all_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 0.0013839517433066354,
-    "train_runtime": 6310.7638,
-    "train_samples": 2101,
-    "train_samples_per_second": 0.333,
-    "train_steps_per_second": 0.007
 }

 {
     "total_flos": 0.0,
+    "train_loss": 0.0014987390906618977,
+    "train_runtime": 3075.3471,
+    "train_samples": 1315,
+    "train_samples_per_second": 0.428,
+    "train_steps_per_second": 0.009
 }

train_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "total_flos": 0.0,
-    "train_loss": 0.0013839517433066354,
-    "train_runtime": 6310.7638,
-    "train_samples": 2101,
-    "train_samples_per_second": 0.333,
-    "train_steps_per_second": 0.007
 }

 {
     "total_flos": 0.0,
+    "train_loss": 0.0014987390906618977,
+    "train_runtime": 3075.3471,
+    "train_samples": 1315,
+    "train_samples_per_second": 0.428,
+    "train_steps_per_second": 0.009
 }

trainer_state.json CHANGED Viewed

@@ -2,9 +2,9 @@
   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
-  "epoch": 0.9814550641940085,
   "eval_steps": 500,
-  "global_step": 43,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
@@ -16,24 +16,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1002.0,
-      "completions/max_terminated_length": 1002.0,
-      "completions/mean_length": 498.02606201171875,
-      "completions/mean_terminated_length": 498.02606201171875,
-      "completions/min_length": 197.0,
-      "completions/min_terminated_length": 197.0,
-      "entropy": 0.079681396484375,
-      "epoch": 0.02282453637660485,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.030818719416856766,
-      "kl": 0.00023728609085083008,
       "learning_rate": 0.0,
-      "loss": 0.0035,
-      "num_tokens": 205147.0,
-      "reward": 0.6144722104072571,
-      "reward_std": 0.13528043031692505,
-      "rewards/cross_solution_unittest/mean": 0.6144722104072571,
-      "rewards/cross_solution_unittest/std": 0.23395827412605286,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 1
@@ -44,25 +44,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 909.0,
-      "completions/mean_length": 530.96875,
-      "completions/mean_terminated_length": 514.9843139648438,
-      "completions/min_length": 277.0,
-      "completions/min_terminated_length": 277.0,
-      "entropy": 0.074462890625,
-      "epoch": 0.0456490727532097,
-      "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.034324560314416885,
-      "kl": 0.0002474188804626465,
-      "learning_rate": 4.000000000000001e-06,
-      "loss": -0.0144,
-      "num_tokens": 421897.0,
-      "reward": 0.654969334602356,
-      "reward_std": 0.14113867282867432,
-      "rewards/cross_solution_unittest/mean": 0.6549693942070007,
-      "rewards/cross_solution_unittest/std": 0.25076690316200256,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 2
@@ -74,24 +74,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1066.0,
-      "completions/max_terminated_length": 1066.0,
-      "completions/mean_length": 495.0364685058594,
-      "completions/mean_terminated_length": 495.0364685058594,
-      "completions/min_length": 285.0,
-      "completions/min_terminated_length": 285.0,
-      "entropy": 0.081207275390625,
-      "epoch": 0.06847360912981455,
-      "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.03333599492907524,
-      "kl": 0.0002630949020385742,
-      "learning_rate": 8.000000000000001e-06,
-      "loss": -0.0036,
-      "num_tokens": 623626.0,
-      "reward": 0.608443558216095,
-      "reward_std": 0.1580158770084381,
-      "rewards/cross_solution_unittest/mean": 0.608443558216095,
-      "rewards/cross_solution_unittest/std": 0.25614744424819946,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 3
@@ -102,25 +102,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1896.0,
-      "completions/max_terminated_length": 1896.0,
-      "completions/mean_length": 531.96875,
-      "completions/mean_terminated_length": 531.96875,
-      "completions/min_length": 290.0,
-      "completions/min_terminated_length": 290.0,
-      "entropy": 0.0734405517578125,
-      "epoch": 0.0912981455064194,
-      "frac_reward_zero_std": 0.1041666716337204,
-      "grad_norm": 0.028182553127408028,
-      "kl": 0.00024580955505371094,
-      "learning_rate": 1.2e-05,
-      "loss": 0.002,
-      "num_tokens": 841736.0,
-      "reward": 0.6449186205863953,
-      "reward_std": 0.12523746490478516,
-      "rewards/cross_solution_unittest/mean": 0.6449186205863953,
-      "rewards/cross_solution_unittest/std": 0.24250361323356628,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 4
@@ -132,24 +132,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 817.0,
-      "completions/max_terminated_length": 817.0,
-      "completions/mean_length": 503.54168701171875,
-      "completions/mean_terminated_length": 503.54168701171875,
-      "completions/min_length": 285.0,
-      "completions/min_terminated_length": 285.0,
-      "entropy": 0.08160400390625,
-      "epoch": 0.11412268188302425,
       "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.03192414715886116,
-      "kl": 0.00025349855422973633,
-      "learning_rate": 1.6000000000000003e-05,
-      "loss": -0.0035,
-      "num_tokens": 1039752.0,
-      "reward": 0.6516736745834351,
-      "reward_std": 0.13371367752552032,
-      "rewards/cross_solution_unittest/mean": 0.6516736149787903,
-      "rewards/cross_solution_unittest/std": 0.25992169976234436,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 5
@@ -162,23 +162,23 @@
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1213.0,
-      "completions/mean_length": 546.5989990234375,
-      "completions/mean_terminated_length": 530.6963500976562,
-      "completions/min_length": 286.0,
-      "completions/min_terminated_length": 286.0,
-      "entropy": 0.075775146484375,
-      "epoch": 0.1369472182596291,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.028952760621905327,
-      "kl": 0.0002371668815612793,
-      "learning_rate": 2e-05,
-      "loss": 0.0251,
-      "num_tokens": 1251119.0,
-      "reward": 0.6305791735649109,
-      "reward_std": 0.11426975578069687,
-      "rewards/cross_solution_unittest/mean": 0.6305791735649109,
-      "rewards/cross_solution_unittest/std": 0.2549576163291931,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 6
@@ -190,24 +190,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1108.0,
-      "completions/max_terminated_length": 1108.0,
-      "completions/mean_length": 509.5520935058594,
-      "completions/mean_terminated_length": 509.5520935058594,
-      "completions/min_length": 293.0,
-      "completions/min_terminated_length": 293.0,
-      "entropy": 0.072845458984375,
-      "epoch": 0.15977175463623394,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.029575008898973465,
-      "kl": 0.00022351741790771484,
-      "learning_rate": 1.9967573081342103e-05,
-      "loss": 0.01,
-      "num_tokens": 1457681.0,
-      "reward": 0.6353155374526978,
-      "reward_std": 0.12137120962142944,
-      "rewards/cross_solution_unittest/mean": 0.635315477848053,
-      "rewards/cross_solution_unittest/std": 0.2490529716014862,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 7
@@ -218,25 +218,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1062.0,
-      "completions/max_terminated_length": 1062.0,
-      "completions/mean_length": 495.3802185058594,
-      "completions/mean_terminated_length": 495.3802185058594,
-      "completions/min_length": 259.0,
-      "completions/min_terminated_length": 259.0,
-      "entropy": 0.08001708984375,
-      "epoch": 0.1825962910128388,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03587636724114418,
-      "kl": 0.00026154518127441406,
-      "learning_rate": 1.9870502626379127e-05,
-      "loss": 0.0011,
-      "num_tokens": 1655898.0,
-      "reward": 0.641265869140625,
-      "reward_std": 0.14582966268062592,
-      "rewards/cross_solution_unittest/mean": 0.6412659287452698,
-      "rewards/cross_solution_unittest/std": 0.24848027527332306,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 8
@@ -247,25 +247,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1454.0,
-      "completions/mean_length": 520.5989990234375,
-      "completions/mean_terminated_length": 504.5602111816406,
-      "completions/min_length": 278.0,
-      "completions/min_terminated_length": 278.0,
-      "entropy": 0.080780029296875,
-      "epoch": 0.20542082738944364,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.03177853301167488,
-      "kl": 0.0002390742301940918,
-      "learning_rate": 1.9709418174260523e-05,
-      "loss": -0.0081,
-      "num_tokens": 1853361.0,
-      "reward": 0.619330883026123,
-      "reward_std": 0.12412238121032715,
-      "rewards/cross_solution_unittest/mean": 0.619330883026123,
-      "rewards/cross_solution_unittest/std": 0.2713783085346222,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 9
@@ -276,25 +276,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1329.0,
-      "completions/mean_length": 519.6979370117188,
-      "completions/mean_terminated_length": 503.6544494628906,
-      "completions/min_length": 278.0,
-      "completions/min_terminated_length": 278.0,
-      "entropy": 0.0826416015625,
-      "epoch": 0.2282453637660485,
-      "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03381970524787903,
-      "kl": 0.0002714395523071289,
-      "learning_rate": 1.9485364419471454e-05,
-      "loss": 0.0137,
-      "num_tokens": 2066775.0,
-      "reward": 0.6427426338195801,
-      "reward_std": 0.12838071584701538,
-      "rewards/cross_solution_unittest/mean": 0.6427426338195801,
-      "rewards/cross_solution_unittest/std": 0.2155393809080124,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 10
@@ -306,24 +306,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1370.0,
-      "completions/max_terminated_length": 1370.0,
-      "completions/mean_length": 531.46875,
-      "completions/mean_terminated_length": 531.46875,
-      "completions/min_length": 263.0,
-      "completions/min_terminated_length": 263.0,
-      "entropy": 0.075469970703125,
-      "epoch": 0.25106990014265335,
-      "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.031015004962682724,
-      "kl": 0.0002269148826599121,
-      "learning_rate": 1.9199794436588244e-05,
-      "loss": -0.0036,
-      "num_tokens": 2280293.0,
-      "reward": 0.6074298620223999,
-      "reward_std": 0.1418527215719223,
-      "rewards/cross_solution_unittest/mean": 0.6074298024177551,
-      "rewards/cross_solution_unittest/std": 0.23763521015644073,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 11
@@ -335,24 +335,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1345.0,
-      "completions/max_terminated_length": 1345.0,
-      "completions/mean_length": 515.9427490234375,
-      "completions/mean_terminated_length": 515.9427490234375,
-      "completions/min_length": 264.0,
-      "completions/min_terminated_length": 264.0,
-      "entropy": 0.07843017578125,
-      "epoch": 0.2738944365192582,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03265320137143135,
-      "kl": 0.00024265050888061523,
-      "learning_rate": 1.8854560256532098e-05,
-      "loss": -0.0093,
-      "num_tokens": 2482738.0,
-      "reward": 0.6170901656150818,
-      "reward_std": 0.13965089619159698,
-      "rewards/cross_solution_unittest/mean": 0.6170901656150818,
-      "rewards/cross_solution_unittest/std": 0.23102088272571564,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 12
@@ -364,24 +364,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1495.0,
-      "completions/max_terminated_length": 1495.0,
-      "completions/mean_length": 510.83856201171875,
-      "completions/mean_terminated_length": 510.83856201171875,
-      "completions/min_length": 259.0,
-      "completions/min_terminated_length": 259.0,
-      "entropy": 0.083984375,
-      "epoch": 0.2967189728958631,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.028935322538018227,
-      "kl": 0.0002440810203552246,
-      "learning_rate": 1.845190085543795e-05,
-      "loss": 0.0174,
-      "num_tokens": 2692567.0,
-      "reward": 0.666698694229126,
-      "reward_std": 0.1532619148492813,
-      "rewards/cross_solution_unittest/mean": 0.666698694229126,
-      "rewards/cross_solution_unittest/std": 0.24854493141174316,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 13
@@ -393,24 +393,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1106.0,
-      "completions/max_terminated_length": 1106.0,
-      "completions/mean_length": 507.703125,
-      "completions/mean_terminated_length": 507.703125,
-      "completions/min_length": 274.0,
-      "completions/min_terminated_length": 274.0,
-      "entropy": 0.075469970703125,
-      "epoch": 0.3195435092724679,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.029261857271194458,
-      "kl": 0.0002554655075073242,
-      "learning_rate": 1.7994427634035016e-05,
-      "loss": 0.0053,
-      "num_tokens": 2897138.0,
-      "reward": 0.6442455053329468,
-      "reward_std": 0.14492110908031464,
-      "rewards/cross_solution_unittest/mean": 0.644245445728302,
-      "rewards/cross_solution_unittest/std": 0.2409852296113968,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 14
@@ -422,24 +422,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 792.0,
-      "completions/max_terminated_length": 792.0,
-      "completions/mean_length": 493.61981201171875,
-      "completions/mean_terminated_length": 493.61981201171875,
-      "completions/min_length": 228.0,
-      "completions/min_terminated_length": 228.0,
-      "entropy": 0.078094482421875,
-      "epoch": 0.34236804564907275,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.030115898698568344,
-      "kl": 0.0002530813217163086,
-      "learning_rate": 1.7485107481711014e-05,
-      "loss": -0.0171,
-      "num_tokens": 3092147.0,
-      "reward": 0.6365131139755249,
-      "reward_std": 0.13549235463142395,
-      "rewards/cross_solution_unittest/mean": 0.6365130543708801,
-      "rewards/cross_solution_unittest/std": 0.22583062946796417,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 15
@@ -451,24 +451,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1203.0,
-      "completions/max_terminated_length": 1203.0,
-      "completions/mean_length": 497.91668701171875,
-      "completions/mean_terminated_length": 497.91668701171875,
-      "completions/min_length": 217.0,
-      "completions/min_terminated_length": 217.0,
-      "entropy": 0.080108642578125,
-      "epoch": 0.3651925820256776,
       "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.030721083283424377,
-      "kl": 0.0002732276916503906,
-      "learning_rate": 1.6927243535095995e-05,
-      "loss": 0.0074,
-      "num_tokens": 3280925.0,
-      "reward": 0.6425144672393799,
-      "reward_std": 0.1044374331831932,
-      "rewards/cross_solution_unittest/mean": 0.6425144672393799,
-      "rewards/cross_solution_unittest/std": 0.2251323163509369,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 16
@@ -479,25 +479,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 920.0,
-      "completions/mean_length": 515.5208740234375,
-      "completions/mean_terminated_length": 499.45550537109375,
-      "completions/min_length": 268.0,
-      "completions/min_terminated_length": 268.0,
-      "entropy": 0.078399658203125,
-      "epoch": 0.3880171184022825,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.031782034784555435,
-      "kl": 0.0002703666687011719,
-      "learning_rate": 1.6324453755953772e-05,
-      "loss": 0.0015,
-      "num_tokens": 3495343.0,
-      "reward": 0.6392027139663696,
-      "reward_std": 0.14693015813827515,
-      "rewards/cross_solution_unittest/mean": 0.6392026543617249,
-      "rewards/cross_solution_unittest/std": 0.2498168796300888,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 17
@@ -510,23 +510,23 @@
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 933.0,
-      "completions/mean_length": 528.8229370117188,
-      "completions/mean_terminated_length": 512.8272094726562,
-      "completions/min_length": 250.0,
-      "completions/min_terminated_length": 250.0,
-      "entropy": 0.077301025390625,
-      "epoch": 0.4108416547788873,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.030714601278305054,
-      "kl": 0.0002853274345397949,
-      "learning_rate": 1.568064746731156e-05,
-      "loss": 0.008,
-      "num_tokens": 3705287.0,
-      "reward": 0.669526994228363,
-      "reward_std": 0.11907248198986053,
-      "rewards/cross_solution_unittest/mean": 0.669526994228363,
-      "rewards/cross_solution_unittest/std": 0.24544787406921387,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 18
@@ -538,24 +538,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 835.0,
-      "completions/max_terminated_length": 835.0,
-      "completions/mean_length": 504.3958435058594,
-      "completions/mean_terminated_length": 504.3958435058594,
-      "completions/min_length": 260.0,
-      "completions/min_terminated_length": 260.0,
-      "entropy": 0.0756072998046875,
-      "epoch": 0.43366619115549215,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.02980155311524868,
-      "kl": 0.0002665519714355469,
-      "learning_rate": 1.5000000000000002e-05,
-      "loss": 0.0059,
-      "num_tokens": 3905667.0,
-      "reward": 0.6205214858055115,
-      "reward_std": 0.13126328587532043,
-      "rewards/cross_solution_unittest/mean": 0.6205214262008667,
-      "rewards/cross_solution_unittest/std": 0.2545018494129181,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 19
@@ -566,25 +566,25 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 2721.0,
-      "completions/mean_length": 536.8489990234375,
-      "completions/mean_terminated_length": 520.895263671875,
-      "completions/min_length": 303.0,
-      "completions/min_terminated_length": 303.0,
-      "entropy": 0.0748443603515625,
-      "epoch": 0.456490727532097,
-      "frac_reward_zero_std": 0.1041666716337204,
-      "grad_norm": 0.029254654422402382,
-      "kl": 0.00030684471130371094,
-      "learning_rate": 1.4286925614030542e-05,
-      "loss": 0.0113,
-      "num_tokens": 4121990.0,
-      "reward": 0.6305307745933533,
-      "reward_std": 0.12758219242095947,
-      "rewards/cross_solution_unittest/mean": 0.6305307745933533,
-      "rewards/cross_solution_unittest/std": 0.24866069853305817,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 20
@@ -596,24 +596,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1022.0,
-      "completions/max_terminated_length": 1022.0,
-      "completions/mean_length": 527.5573120117188,
-      "completions/mean_terminated_length": 527.5573120117188,
-      "completions/min_length": 252.0,
-      "completions/min_terminated_length": 252.0,
-      "entropy": 0.0727691650390625,
-      "epoch": 0.4793152639087018,
       "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.028263313695788383,
-      "kl": 0.00026792287826538086,
-      "learning_rate": 1.3546048870425356e-05,
-      "loss": 0.0059,
-      "num_tokens": 4330363.0,
-      "reward": 0.6157366633415222,
-      "reward_std": 0.14285579323768616,
-      "rewards/cross_solution_unittest/mean": 0.6157366037368774,
-      "rewards/cross_solution_unittest/std": 0.24484053254127502,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 21
@@ -625,24 +625,24 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1639.0,
-      "completions/max_terminated_length": 1639.0,
-      "completions/mean_length": 497.328125,
-      "completions/mean_terminated_length": 497.328125,
-      "completions/min_length": 277.0,
-      "completions/min_terminated_length": 277.0,
-      "entropy": 0.080413818359375,
-      "epoch": 0.5021398002853067,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03302839770913124,
-      "kl": 0.0003446340560913086,
-      "learning_rate": 1.2782174639164528e-05,
-      "loss": 0.0044,
-      "num_tokens": 4529370.0,
-      "reward": 0.6745923757553101,
-      "reward_std": 0.14387564361095428,
-      "rewards/cross_solution_unittest/mean": 0.6745923161506653,
-      "rewards/cross_solution_unittest/std": 0.2329888641834259,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 22
@@ -654,317 +654,27 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 908.0,
-      "completions/max_terminated_length": 908.0,
-      "completions/mean_length": 495.74481201171875,
-      "completions/mean_terminated_length": 495.74481201171875,
-      "completions/min_length": 247.0,
-      "completions/min_terminated_length": 247.0,
-      "entropy": 0.07891845703125,
-      "epoch": 0.5249643366619116,
-      "frac_reward_zero_std": 0.125,
-      "grad_norm": 0.031968750059604645,
-      "kl": 0.0003135800361633301,
-      "learning_rate": 1.2000256937760446e-05,
-      "loss": 0.0035,
-      "num_tokens": 4734861.0,
-      "reward": 0.6773189902305603,
-      "reward_std": 0.11135952174663544,
-      "rewards/cross_solution_unittest/mean": 0.6773189902305603,
-      "rewards/cross_solution_unittest/std": 0.24014313519001007,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 23
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 966.0,
-      "completions/max_terminated_length": 966.0,
-      "completions/mean_length": 487.16668701171875,
-      "completions/mean_terminated_length": 487.16668701171875,
-      "completions/min_length": 270.0,
-      "completions/min_terminated_length": 270.0,
-      "entropy": 0.0809326171875,
-      "epoch": 0.5477888730385164,
-      "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.033082712441682816,
-      "kl": 0.00030118227005004883,
-      "learning_rate": 1.1205366802553231e-05,
-      "loss": -0.0181,
-      "num_tokens": 4929177.0,
-      "reward": 0.5979318618774414,
-      "reward_std": 0.1381441205739975,
-      "rewards/cross_solution_unittest/mean": 0.5979318618774414,
-      "rewards/cross_solution_unittest/std": 0.2676696479320526,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 24
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1080.0,
-      "completions/max_terminated_length": 1080.0,
-      "completions/mean_length": 499.63543701171875,
-      "completions/mean_terminated_length": 499.63543701171875,
-      "completions/min_length": 242.0,
-      "completions/min_terminated_length": 242.0,
-      "entropy": 0.0802001953125,
-      "epoch": 0.5706134094151213,
-      "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.0314960740506649,
-      "kl": 0.00029587745666503906,
-      "learning_rate": 1.0402659401094154e-05,
-      "loss": 0.0047,
-      "num_tokens": 5126741.0,
-      "reward": 0.6157103776931763,
-      "reward_std": 0.12421576678752899,
-      "rewards/cross_solution_unittest/mean": 0.615710437297821,
-      "rewards/cross_solution_unittest/std": 0.21989282965660095,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 25
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.9375,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 852.0,
-      "completions/mean_length": 536.3385620117188,
-      "completions/mean_terminated_length": 504.2579040527344,
-      "completions/min_length": 285.0,
-      "completions/min_terminated_length": 285.0,
-      "entropy": 0.07733154296875,
-      "epoch": 0.5934379457917262,
-      "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03292542323470116,
-      "kl": 0.0003018975257873535,
-      "learning_rate": 9.597340598905851e-06,
-      "loss": 0.0109,
-      "num_tokens": 5332770.0,
-      "reward": 0.614662766456604,
-      "reward_std": 0.13745400309562683,
-      "rewards/cross_solution_unittest/mean": 0.6146627068519592,
-      "rewards/cross_solution_unittest/std": 0.24220500886440277,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 26
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 924.0,
-      "completions/max_terminated_length": 924.0,
-      "completions/mean_length": 505.734375,
-      "completions/mean_terminated_length": 505.734375,
-      "completions/min_length": 268.0,
-      "completions/min_terminated_length": 268.0,
-      "entropy": 0.0759124755859375,
-      "epoch": 0.6162624821683309,
-      "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.02907959185540676,
-      "kl": 0.0002906322479248047,
-      "learning_rate": 8.79463319744677e-06,
-      "loss": -0.0097,
-      "num_tokens": 5542205.0,
-      "reward": 0.6280584931373596,
-      "reward_std": 0.1270303875207901,
-      "rewards/cross_solution_unittest/mean": 0.6280584931373596,
-      "rewards/cross_solution_unittest/std": 0.25405508279800415,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 27
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 902.0,
-      "completions/max_terminated_length": 902.0,
-      "completions/mean_length": 504.6302185058594,
-      "completions/mean_terminated_length": 504.6302185058594,
-      "completions/min_length": 225.0,
-      "completions/min_terminated_length": 225.0,
-      "entropy": 0.081817626953125,
-      "epoch": 0.6390870185449358,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.03565341234207153,
-      "kl": 0.0003236532211303711,
-      "learning_rate": 7.999743062239557e-06,
-      "loss": 0.0158,
-      "num_tokens": 5747370.0,
-      "reward": 0.6250073909759521,
-      "reward_std": 0.13367865979671478,
-      "rewards/cross_solution_unittest/mean": 0.6250073313713074,
-      "rewards/cross_solution_unittest/std": 0.23361748456954956,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 28
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 970.0,
-      "completions/mean_length": 520.234375,
-      "completions/mean_terminated_length": 504.1937255859375,
-      "completions/min_length": 260.0,
-      "completions/min_terminated_length": 260.0,
-      "entropy": 0.073333740234375,
-      "epoch": 0.6619115549215406,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.03009534440934658,
-      "kl": 0.00029665231704711914,
-      "learning_rate": 7.217825360835475e-06,
-      "loss": -0.0107,
-      "num_tokens": 5952477.0,
-      "reward": 0.6348472833633423,
-      "reward_std": 0.13845080137252808,
-      "rewards/cross_solution_unittest/mean": 0.6348472237586975,
-      "rewards/cross_solution_unittest/std": 0.24842146039009094,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 29
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1245.0,
-      "completions/mean_length": 520.515625,
-      "completions/mean_terminated_length": 504.4764404296875,
       "completions/min_length": 243.0,
       "completions/min_terminated_length": 243.0,
-      "entropy": 0.079833984375,
-      "epoch": 0.6847360912981455,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.0347207672894001,
-      "kl": 0.0003616809844970703,
-      "learning_rate": 6.453951129574644e-06,
-      "loss": 0.0177,
-      "num_tokens": 6162950.0,
-      "reward": 0.6565889120101929,
-      "reward_std": 0.16731536388397217,
-      "rewards/cross_solution_unittest/mean": 0.6565888524055481,
-      "rewards/cross_solution_unittest/std": 0.27589210867881775,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 30
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1040.0,
-      "completions/mean_length": 533.5989990234375,
-      "completions/mean_terminated_length": 517.6282958984375,
-      "completions/min_length": 224.0,
-      "completions/min_terminated_length": 224.0,
-      "entropy": 0.07684326171875,
-      "epoch": 0.7075606276747504,
       "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.0333818718791008,
-      "kl": 0.00035303831100463867,
-      "learning_rate": 5.713074385969457e-06,
-      "loss": -0.0013,
-      "num_tokens": 6378077.0,
-      "reward": 0.6434895992279053,
-      "reward_std": 0.14427071809768677,
-      "rewards/cross_solution_unittest/mean": 0.6434895992279053,
-      "rewards/cross_solution_unittest/std": 0.23883987963199615,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 31
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1025.0,
-      "completions/max_terminated_length": 1025.0,
-      "completions/mean_length": 500.1875,
-      "completions/mean_terminated_length": 500.1875,
-      "completions/min_length": 251.0,
-      "completions/min_terminated_length": 251.0,
-      "entropy": 0.078643798828125,
-      "epoch": 0.7303851640513552,
-      "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.031248383224010468,
-      "kl": 0.00033414363861083984,
-      "learning_rate": 5.000000000000003e-06,
-      "loss": 0.0005,
-      "num_tokens": 6583735.0,
-      "reward": 0.6346912384033203,
-      "reward_std": 0.11402349174022675,
-      "rewards/cross_solution_unittest/mean": 0.6346912384033203,
-      "rewards/cross_solution_unittest/std": 0.23955823481082916,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 32
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1642.0,
-      "completions/mean_length": 526.59375,
-      "completions/mean_terminated_length": 510.5863952636719,
-      "completions/min_length": 221.0,
-      "completions/min_terminated_length": 221.0,
-      "entropy": 0.077301025390625,
-      "epoch": 0.7532097004279601,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.032974906265735626,
-      "kl": 0.000325620174407959,
-      "learning_rate": 4.319352532688444e-06,
-      "loss": -0.018,
-      "num_tokens": 6798257.0,
-      "reward": 0.6666222810745239,
-      "reward_std": 0.1389816403388977,
-      "rewards/cross_solution_unittest/mean": 0.6666222214698792,
-      "rewards/cross_solution_unittest/std": 0.24471229314804077,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
-      "step": 33
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -972,144 +682,28 @@
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 962.0,
-      "completions/mean_length": 521.9479370117188,
-      "completions/mean_terminated_length": 505.9162292480469,
-      "completions/min_length": 239.0,
-      "completions/min_terminated_length": 239.0,
-      "entropy": 0.0804443359375,
-      "epoch": 0.776034236804565,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.03674512356519699,
-      "kl": 0.0004087686538696289,
-      "learning_rate": 3.6755462440462288e-06,
-      "loss": 0.0082,
-      "num_tokens": 7010023.0,
-      "reward": 0.6678439974784851,
-      "reward_std": 0.1117190346121788,
-      "rewards/cross_solution_unittest/mean": 0.6678439974784851,
-      "rewards/cross_solution_unittest/std": 0.24756233394145966,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 34
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 995.0,
-      "completions/max_terminated_length": 995.0,
-      "completions/mean_length": 519.1146240234375,
-      "completions/mean_terminated_length": 519.1146240234375,
-      "completions/min_length": 237.0,
-      "completions/min_terminated_length": 237.0,
-      "entropy": 0.076324462890625,
-      "epoch": 0.7988587731811697,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.029553759843111038,
-      "kl": 0.0003510117530822754,
-      "learning_rate": 3.0727564649040066e-06,
-      "loss": 0.0151,
-      "num_tokens": 7216697.0,
-      "reward": 0.6525782346725464,
-      "reward_std": 0.15372397005558014,
-      "rewards/cross_solution_unittest/mean": 0.6525782346725464,
-      "rewards/cross_solution_unittest/std": 0.24588163197040558,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 35
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -5.0,
-      "completions/max_length": 1043.0,
-      "completions/max_terminated_length": 1043.0,
-      "completions/mean_length": 492.859375,
-      "completions/mean_terminated_length": 492.859375,
-      "completions/min_length": 266.0,
-      "completions/min_terminated_length": 266.0,
-      "entropy": 0.07861328125,
-      "epoch": 0.8216833095577746,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.030761953443288803,
-      "kl": 0.00032657384872436523,
-      "learning_rate": 2.514892518288988e-06,
-      "loss": 0.0029,
-      "num_tokens": 7411044.0,
-      "reward": 0.6515220999717712,
-      "reward_std": 0.1183708906173706,
-      "rewards/cross_solution_unittest/mean": 0.6515220999717712,
-      "rewards/cross_solution_unittest/std": 0.2343326210975647,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 36
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 829.0,
-      "completions/mean_length": 524.6041870117188,
-      "completions/mean_terminated_length": 508.5863952636719,
-      "completions/min_length": 265.0,
-      "completions/min_terminated_length": 265.0,
-      "entropy": 0.076904296875,
-      "epoch": 0.8445078459343794,
-      "frac_reward_zero_std": 0.0625,
-      "grad_norm": 0.030993396416306496,
-      "kl": 0.00032192468643188477,
-      "learning_rate": 2.0055723659649907e-06,
-      "loss": -0.0032,
-      "num_tokens": 7628810.0,
-      "reward": 0.7275886535644531,
-      "reward_std": 0.12316948175430298,
-      "rewards/cross_solution_unittest/mean": 0.7275886535644531,
-      "rewards/cross_solution_unittest/std": 0.21457894146442413,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
-      "step": 37
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 1203.0,
-      "completions/mean_length": 534.2708740234375,
-      "completions/mean_terminated_length": 518.3036499023438,
-      "completions/min_length": 239.0,
-      "completions/min_terminated_length": 239.0,
-      "entropy": 0.075714111328125,
-      "epoch": 0.8673323823109843,
-      "frac_reward_zero_std": 0.0833333358168602,
-      "grad_norm": 0.02853851392865181,
-      "kl": 0.0003237724304199219,
-      "learning_rate": 1.5480991445620541e-06,
-      "loss": -0.0054,
-      "num_tokens": 7843208.0,
-      "reward": 0.6594494581222534,
-      "reward_std": 0.10400386154651642,
-      "rewards/cross_solution_unittest/mean": 0.6594493985176086,
-      "rewards/cross_solution_unittest/std": 0.2267450988292694,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 38
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -1118,85 +712,27 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 936.0,
-      "completions/max_terminated_length": 936.0,
-      "completions/mean_length": 515.4479370117188,
-      "completions/mean_terminated_length": 515.4479370117188,
-      "completions/min_length": 292.0,
-      "completions/min_terminated_length": 292.0,
-      "entropy": 0.0770721435546875,
-      "epoch": 0.8901569186875892,
       "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.030984513461589813,
-      "kl": 0.0003541111946105957,
-      "learning_rate": 1.1454397434679022e-06,
-      "loss": 0.0019,
-      "num_tokens": 8059010.0,
-      "reward": 0.682985782623291,
-      "reward_std": 0.13432684540748596,
-      "rewards/cross_solution_unittest/mean": 0.682985782623291,
-      "rewards/cross_solution_unittest/std": 0.2116367071866989,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
-      "step": 39
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.9375,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 3309.0,
-      "completions/mean_length": 566.5,
-      "completions/mean_terminated_length": 534.7368774414062,
-      "completions/min_length": 288.0,
-      "completions/min_terminated_length": 288.0,
-      "entropy": 0.07586669921875,
-      "epoch": 0.912981455064194,
-      "frac_reward_zero_std": 0.0416666679084301,
-      "grad_norm": 0.03089836984872818,
-      "kl": 0.0003343820571899414,
-      "learning_rate": 8.002055634117578e-07,
-      "loss": -0.0147,
-      "num_tokens": 8270756.0,
-      "reward": 0.644805908203125,
-      "reward_std": 0.12590646743774414,
-      "rewards/cross_solution_unittest/mean": 0.6448058485984802,
-      "rewards/cross_solution_unittest/std": 0.2504419982433319,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 40
-    },
-    {
-      "clip_ratio/high_max": 0.0,
-      "clip_ratio/high_mean": 0.0,
-      "clip_ratio/low_mean": 0.0,
-      "clip_ratio/low_min": 0.0,
-      "clip_ratio/region_mean": 0.0,
-      "completions/clipped_ratio": -4.96875,
-      "completions/max_length": 3584.0,
-      "completions/max_terminated_length": 871.0,
-      "completions/mean_length": 526.234375,
-      "completions/mean_terminated_length": 510.2251281738281,
-      "completions/min_length": 290.0,
-      "completions/min_terminated_length": 290.0,
-      "entropy": 0.074859619140625,
-      "epoch": 0.9358059914407989,
-      "frac_reward_zero_std": 0.0,
-      "grad_norm": 0.03084583766758442,
-      "kl": 0.0003114938735961914,
-      "learning_rate": 5.146355805285452e-07,
-      "loss": -0.0036,
-      "num_tokens": 8480439.0,
-      "reward": 0.6170955896377563,
-      "reward_std": 0.11663064360618591,
-      "rewards/cross_solution_unittest/mean": 0.6170955300331116,
-      "rewards/cross_solution_unittest/std": 0.21888427436351776,
-      "rewards/format_reward/mean": 0.0,
-      "rewards/format_reward/std": 0.0,
-      "step": 41
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -1205,27 +741,27 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 911.0,
-      "completions/max_terminated_length": 911.0,
-      "completions/mean_length": 512.2864990234375,
-      "completions/mean_terminated_length": 512.2864990234375,
-      "completions/min_length": 271.0,
-      "completions/min_terminated_length": 271.0,
-      "entropy": 0.076995849609375,
-      "epoch": 0.9586305278174037,
       "frac_reward_zero_std": 0.02083333395421505,
-      "grad_norm": 0.031123578548431396,
-      "kl": 0.0003229379653930664,
-      "learning_rate": 2.905818257394799e-07,
-      "loss": -0.0134,
-      "num_tokens": 8687490.0,
-      "reward": 0.6666769981384277,
-      "reward_std": 0.13320571184158325,
-      "rewards/cross_solution_unittest/mean": 0.6666769981384277,
-      "rewards/cross_solution_unittest/std": 0.23345570266246796,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
-      "step": 42
     },
     {
       "clip_ratio/high_max": 0.0,
@@ -1234,41 +770,41 @@
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
-      "completions/max_length": 902.0,
-      "completions/max_terminated_length": 902.0,
-      "completions/mean_length": 488.18231201171875,
-      "completions/mean_terminated_length": 488.18231201171875,
-      "completions/min_length": 290.0,
-      "completions/min_terminated_length": 290.0,
-      "entropy": 0.082733154296875,
-      "epoch": 0.9814550641940085,
-      "frac_reward_zero_std": 0.1041666716337204,
-      "grad_norm": 0.03318248316645622,
-      "kl": 0.00038182735443115234,
-      "learning_rate": 1.2949737362087156e-07,
       "loss": 0.0135,
-      "num_tokens": 8885881.0,
-      "reward": 0.6386904120445251,
-      "reward_std": 0.12852439284324646,
-      "rewards/cross_solution_unittest/mean": 0.6386904120445251,
-      "rewards/cross_solution_unittest/std": 0.24438685178756714,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
-      "step": 43
     },
     {
-      "epoch": 0.9814550641940085,
-      "step": 43,
       "total_flos": 0.0,
-      "train_loss": 0.0013839517433066354,
-      "train_runtime": 6310.7638,
-      "train_samples_per_second": 0.333,
-      "train_steps_per_second": 0.007
     }
   ],
   "logging_steps": 1,
-  "max_steps": 44,
-  "num_input_tokens_seen": 8885881,
   "num_train_epochs": 1,
   "save_steps": 50,
   "stateful_callbacks": {

   "best_global_step": null,
   "best_metric": null,
   "best_model_checkpoint": null,
+  "epoch": 0.9840546697038725,
   "eval_steps": 500,
+  "global_step": 27,
   "is_hyper_param_search": false,
   "is_local_process_zero": true,
   "is_world_process_zero": true,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1030.0,
+      "completions/max_terminated_length": 1030.0,
+      "completions/mean_length": 520.4375,
+      "completions/mean_terminated_length": 520.4375,
+      "completions/min_length": 259.0,
+      "completions/min_terminated_length": 259.0,
+      "entropy": 0.0713653564453125,
+      "epoch": 0.03644646924829157,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.02890011854469776,
+      "kl": 0.00021544098854064941,
       "learning_rate": 0.0,
+      "loss": -0.0031,
+      "num_tokens": 211542.0,
+      "reward": 0.5595489144325256,
+      "reward_std": 0.10752050578594208,
+      "rewards/cross_solution_unittest/mean": 0.5595488548278809,
+      "rewards/cross_solution_unittest/std": 0.18392445147037506,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 1
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -4.90625,
       "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 973.0,
+      "completions/mean_length": 565.9271240234375,
+      "completions/mean_terminated_length": 518.0211181640625,
+      "completions/min_length": 220.0,
+      "completions/min_terminated_length": 220.0,
+      "entropy": 0.0749359130859375,
+      "epoch": 0.07289293849658314,
+      "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.02937042899429798,
+      "kl": 0.00021922588348388672,
+      "learning_rate": 6.666666666666667e-06,
+      "loss": -0.0023,
+      "num_tokens": 432880.0,
+      "reward": 0.616872251033783,
+      "reward_std": 0.10939260572195053,
+      "rewards/cross_solution_unittest/mean": 0.6168721914291382,
+      "rewards/cross_solution_unittest/std": 0.19950935244560242,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 2
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1322.0,
+      "completions/max_terminated_length": 1322.0,
+      "completions/mean_length": 517.7291870117188,
+      "completions/mean_terminated_length": 517.7291870117188,
+      "completions/min_length": 273.0,
+      "completions/min_terminated_length": 273.0,
+      "entropy": 0.0756378173828125,
+      "epoch": 0.10933940774487472,
+      "frac_reward_zero_std": 0.0,
+      "grad_norm": 0.03240806981921196,
+      "kl": 0.00023889541625976562,
+      "learning_rate": 1.3333333333333333e-05,
+      "loss": -0.0129,
+      "num_tokens": 645970.0,
+      "reward": 0.5990354418754578,
+      "reward_std": 0.1072663888335228,
+      "rewards/cross_solution_unittest/mean": 0.5990354418754578,
+      "rewards/cross_solution_unittest/std": 0.19944854080677032,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 3
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -4.96875,
+      "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 1919.0,
+      "completions/mean_length": 533.5052490234375,
+      "completions/mean_terminated_length": 517.5340576171875,
+      "completions/min_length": 212.0,
+      "completions/min_terminated_length": 212.0,
+      "entropy": 0.0742034912109375,
+      "epoch": 0.14578587699316628,
+      "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.02899506501853466,
+      "kl": 0.0002148747444152832,
+      "learning_rate": 2e-05,
+      "loss": -0.0112,
+      "num_tokens": 848277.0,
+      "reward": 0.5434086322784424,
+      "reward_std": 0.09839596599340439,
+      "rewards/cross_solution_unittest/mean": 0.5434086322784424,
+      "rewards/cross_solution_unittest/std": 0.21125413477420807,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 4
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 929.0,
+      "completions/max_terminated_length": 929.0,
+      "completions/mean_length": 512.15625,
+      "completions/mean_terminated_length": 512.15625,
+      "completions/min_length": 241.0,
+      "completions/min_terminated_length": 241.0,
+      "entropy": 0.074676513671875,
+      "epoch": 0.18223234624145787,
       "frac_reward_zero_std": 0.0625,
+      "grad_norm": 0.030133171007037163,
+      "kl": 0.0002295374870300293,
+      "learning_rate": 1.9921147013144782e-05,
+      "loss": -0.0018,
+      "num_tokens": 1058649.0,
+      "reward": 0.5704458951950073,
+      "reward_std": 0.10581853240728378,
+      "rewards/cross_solution_unittest/mean": 0.5704458951950073,
+      "rewards/cross_solution_unittest/std": 0.20132118463516235,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 5
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 1124.0,
+      "completions/mean_length": 533.1354370117188,
+      "completions/mean_terminated_length": 517.1622924804688,
+      "completions/min_length": 265.0,
+      "completions/min_terminated_length": 265.0,
+      "entropy": 0.07135009765625,
+      "epoch": 0.21867881548974943,
+      "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.030490361154079437,
+      "kl": 0.00021791458129882812,
+      "learning_rate": 1.9685831611286312e-05,
+      "loss": 0.0,
+      "num_tokens": 1267713.0,
+      "reward": 0.579592227935791,
+      "reward_std": 0.12655234336853027,
+      "rewards/cross_solution_unittest/mean": 0.579592227935791,
+      "rewards/cross_solution_unittest/std": 0.2343016117811203,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 6
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 880.0,
+      "completions/max_terminated_length": 880.0,
+      "completions/mean_length": 505.27606201171875,
+      "completions/mean_terminated_length": 505.27606201171875,
+      "completions/min_length": 233.0,
+      "completions/min_terminated_length": 233.0,
+      "entropy": 0.07843017578125,
+      "epoch": 0.255125284738041,
+      "frac_reward_zero_std": 0.0,
+      "grad_norm": 0.031553227454423904,
+      "kl": 0.00024181604385375977,
+      "learning_rate": 1.9297764858882516e-05,
+      "loss": 0.0249,
+      "num_tokens": 1467406.0,
+      "reward": 0.5725040435791016,
+      "reward_std": 0.10377390682697296,
+      "rewards/cross_solution_unittest/mean": 0.5725040435791016,
+      "rewards/cross_solution_unittest/std": 0.21926818788051605,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 7
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -4.9375,
+      "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 917.0,
+      "completions/mean_length": 520.34375,
+      "completions/mean_terminated_length": 488.0947570800781,
+      "completions/min_length": 277.0,
+      "completions/min_terminated_length": 277.0,
+      "entropy": 0.0777130126953125,
+      "epoch": 0.29157175398633256,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.030870754271745682,
+      "kl": 0.00022351741790771484,
+      "learning_rate": 1.8763066800438638e-05,
+      "loss": 0.0048,
+      "num_tokens": 1677876.0,
+      "reward": 0.5693823099136353,
+      "reward_std": 0.12342718243598938,
+      "rewards/cross_solution_unittest/mean": 0.5693822503089905,
+      "rewards/cross_solution_unittest/std": 0.22165222465991974,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 8
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1040.0,
+      "completions/max_terminated_length": 1040.0,
+      "completions/mean_length": 500.7864685058594,
+      "completions/mean_terminated_length": 500.7864685058594,
+      "completions/min_length": 308.0,
+      "completions/min_terminated_length": 308.0,
+      "entropy": 0.07794189453125,
+      "epoch": 0.32801822323462415,
+      "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.030629878863692284,
+      "kl": 0.00024515390396118164,
+      "learning_rate": 1.8090169943749477e-05,
+      "loss": 0.0083,
+      "num_tokens": 1880643.0,
+      "reward": 0.6052049398422241,
+      "reward_std": 0.109284907579422,
+      "rewards/cross_solution_unittest/mean": 0.6052048802375793,
+      "rewards/cross_solution_unittest/std": 0.20577581226825714,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 9
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -5.0,
+      "completions/max_length": 945.0,
+      "completions/max_terminated_length": 945.0,
+      "completions/mean_length": 495.9114685058594,
+      "completions/mean_terminated_length": 495.9114685058594,
+      "completions/min_length": 237.0,
+      "completions/min_terminated_length": 237.0,
+      "entropy": 0.07415771484375,
+      "epoch": 0.36446469248291574,
+      "frac_reward_zero_std": 0.0625,
+      "grad_norm": 0.032062605023384094,
+      "kl": 0.0002478361129760742,
+      "learning_rate": 1.7289686274214116e-05,
+      "loss": -0.0011,
+      "num_tokens": 2071294.0,
+      "reward": 0.5925840735435486,
+      "reward_std": 0.11311423778533936,
+      "rewards/cross_solution_unittest/mean": 0.5925840735435486,
+      "rewards/cross_solution_unittest/std": 0.19814850389957428,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 10
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1178.0,
+      "completions/max_terminated_length": 1178.0,
+      "completions/mean_length": 509.1614685058594,
+      "completions/mean_terminated_length": 509.1614685058594,
+      "completions/min_length": 270.0,
+      "completions/min_terminated_length": 270.0,
+      "entropy": 0.073944091796875,
+      "epoch": 0.4009111617312073,
+      "frac_reward_zero_std": 0.0625,
+      "grad_norm": 0.030858976766467094,
+      "kl": 0.00023108720779418945,
+      "learning_rate": 1.63742398974869e-05,
+      "loss": 0.002,
+      "num_tokens": 2273865.0,
+      "reward": 0.5973134636878967,
+      "reward_std": 0.10415109246969223,
+      "rewards/cross_solution_unittest/mean": 0.5973134636878967,
+      "rewards/cross_solution_unittest/std": 0.21376842260360718,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 11
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 859.0,
+      "completions/max_terminated_length": 859.0,
+      "completions/mean_length": 487.49481201171875,
+      "completions/mean_terminated_length": 487.49481201171875,
+      "completions/min_length": 259.0,
+      "completions/min_terminated_length": 259.0,
+      "entropy": 0.081573486328125,
+      "epoch": 0.43735763097949887,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.035162851214408875,
+      "kl": 0.00028896331787109375,
+      "learning_rate": 1.5358267949789968e-05,
+      "loss": 0.0162,
+      "num_tokens": 2472554.0,
+      "reward": 0.5868850946426392,
+      "reward_std": 0.11655251681804657,
+      "rewards/cross_solution_unittest/mean": 0.5868851542472839,
+      "rewards/cross_solution_unittest/std": 0.2154010534286499,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 12
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1389.0,
+      "completions/max_terminated_length": 1389.0,
+      "completions/mean_length": 529.6041870117188,
+      "completions/mean_terminated_length": 529.6041870117188,
+      "completions/min_length": 217.0,
+      "completions/min_terminated_length": 217.0,
+      "entropy": 0.07177734375,
+      "epoch": 0.47380410022779046,
+      "frac_reward_zero_std": 0.1041666716337204,
+      "grad_norm": 0.029409615322947502,
+      "kl": 0.00025135278701782227,
+      "learning_rate": 1.4257792915650728e-05,
+      "loss": 0.0091,
+      "num_tokens": 2682176.0,
+      "reward": 0.5738339424133301,
+      "reward_std": 0.1161453127861023,
+      "rewards/cross_solution_unittest/mean": 0.5738338828086853,
+      "rewards/cross_solution_unittest/std": 0.22482098639011383,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 13
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 959.0,
+      "completions/max_terminated_length": 959.0,
+      "completions/mean_length": 504.6614685058594,
+      "completions/mean_terminated_length": 504.6614685058594,
+      "completions/min_length": 280.0,
+      "completions/min_terminated_length": 280.0,
+      "entropy": 0.075927734375,
+      "epoch": 0.510250569476082,
+      "frac_reward_zero_std": 0.0,
+      "grad_norm": 0.03111596405506134,
+      "kl": 0.00023740530014038086,
+      "learning_rate": 1.3090169943749475e-05,
+      "loss": -0.0007,
+      "num_tokens": 2885113.0,
+      "reward": 0.5357317328453064,
+      "reward_std": 0.1203121468424797,
+      "rewards/cross_solution_unittest/mean": 0.5357317328453064,
+      "rewards/cross_solution_unittest/std": 0.2490031123161316,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 14
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 926.0,
+      "completions/max_terminated_length": 926.0,
+      "completions/mean_length": 502.9739685058594,
+      "completions/mean_terminated_length": 502.9739685058594,
+      "completions/min_length": 243.0,
+      "completions/min_terminated_length": 243.0,
+      "entropy": 0.08111572265625,
+      "epoch": 0.5466970387243736,
+      "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.031033573672175407,
+      "kl": 0.00027930736541748047,
+      "learning_rate": 1.187381314585725e-05,
+      "loss": -0.0008,
+      "num_tokens": 3098294.0,
+      "reward": 0.5646395683288574,
+      "reward_std": 0.10859303176403046,
+      "rewards/cross_solution_unittest/mean": 0.5646395087242126,
+      "rewards/cross_solution_unittest/std": 0.19871041178703308,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 15
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1039.0,
+      "completions/max_terminated_length": 1039.0,
+      "completions/mean_length": 512.5729370117188,
+      "completions/mean_terminated_length": 512.5729370117188,
+      "completions/min_length": 223.0,
+      "completions/min_terminated_length": 223.0,
+      "entropy": 0.075897216796875,
+      "epoch": 0.5831435079726651,
       "frac_reward_zero_std": 0.0833333358168602,
+      "grad_norm": 0.02894579991698265,
+      "kl": 0.0002409219741821289,
+      "learning_rate": 1.0627905195293135e-05,
+      "loss": 0.0065,
+      "num_tokens": 3304324.0,
+      "reward": 0.5695656538009644,
+      "reward_std": 0.10569658130407333,
+      "rewards/cross_solution_unittest/mean": 0.5695656538009644,
+      "rewards/cross_solution_unittest/std": 0.20740273594856262,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 16
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -5.0,
+      "completions/max_length": 898.0,
+      "completions/max_terminated_length": 898.0,
+      "completions/mean_length": 506.38543701171875,
+      "completions/mean_terminated_length": 506.38543701171875,
+      "completions/min_length": 249.0,
+      "completions/min_terminated_length": 249.0,
+      "entropy": 0.0773773193359375,
+      "epoch": 0.6195899772209568,
+      "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.03125438094139099,
+      "kl": 0.00026804208755493164,
+      "learning_rate": 9.372094804706867e-06,
+      "loss": 0.0019,
+      "num_tokens": 3520562.0,
+      "reward": 0.609571099281311,
+      "reward_std": 0.09919850528240204,
+      "rewards/cross_solution_unittest/mean": 0.6095710396766663,
+      "rewards/cross_solution_unittest/std": 0.20345403254032135,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 17
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -4.96875,
       "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 1040.0,
+      "completions/mean_length": 550.796875,
+      "completions/mean_terminated_length": 534.916259765625,
+      "completions/min_length": 270.0,
+      "completions/min_terminated_length": 270.0,
+      "entropy": 0.072509765625,
+      "epoch": 0.6560364464692483,
+      "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.03002159111201763,
+      "kl": 0.00025850534439086914,
+      "learning_rate": 8.126186854142752e-06,
+      "loss": -0.0039,
+      "num_tokens": 3739721.0,
+      "reward": 0.6008386015892029,
+      "reward_std": 0.11992423236370087,
+      "rewards/cross_solution_unittest/mean": 0.6008385419845581,
+      "rewards/cross_solution_unittest/std": 0.2292270064353943,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 18
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1008.0,
+      "completions/max_terminated_length": 1008.0,
+      "completions/mean_length": 518.234375,
+      "completions/mean_terminated_length": 518.234375,
+      "completions/min_length": 197.0,
+      "completions/min_terminated_length": 197.0,
+      "entropy": 0.0797119140625,
+      "epoch": 0.6924829157175398,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.03135575354099274,
+      "kl": 0.00027000904083251953,
+      "learning_rate": 6.909830056250527e-06,
+      "loss": -0.01,
+      "num_tokens": 3935738.0,
+      "reward": 0.5821563005447388,
+      "reward_std": 0.11166616529226303,
+      "rewards/cross_solution_unittest/mean": 0.582156240940094,
+      "rewards/cross_solution_unittest/std": 0.20721960067749023,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 19
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -4.9375,
       "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 1872.0,
+      "completions/mean_length": 546.28125,
+      "completions/mean_terminated_length": 514.3052978515625,
+      "completions/min_length": 272.0,
+      "completions/min_terminated_length": 272.0,
+      "entropy": 0.07513427734375,
+      "epoch": 0.7289293849658315,
+      "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.03126757591962814,
+      "kl": 0.00025522708892822266,
+      "learning_rate": 5.742207084349274e-06,
+      "loss": -0.0025,
+      "num_tokens": 4145042.0,
+      "reward": 0.5788866281509399,
+      "reward_std": 0.12265484035015106,
+      "rewards/cross_solution_unittest/mean": 0.5788866281509399,
+      "rewards/cross_solution_unittest/std": 0.2369561791419983,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 20
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1437.0,
+      "completions/max_terminated_length": 1437.0,
+      "completions/mean_length": 495.43231201171875,
+      "completions/mean_terminated_length": 495.43231201171875,
+      "completions/min_length": 195.0,
+      "completions/min_terminated_length": 195.0,
+      "entropy": 0.07891845703125,
+      "epoch": 0.765375854214123,
       "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.032083988189697266,
+      "kl": 0.00027364492416381836,
+      "learning_rate": 4.641732050210032e-06,
+      "loss": 0.0086,
+      "num_tokens": 4336333.0,
+      "reward": 0.59117591381073,
+      "reward_std": 0.12750200927257538,
+      "rewards/cross_solution_unittest/mean": 0.5911758542060852,
+      "rewards/cross_solution_unittest/std": 0.22702106833457947,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 21
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 1204.0,
+      "completions/max_terminated_length": 1204.0,
+      "completions/mean_length": 500.3177185058594,
+      "completions/mean_terminated_length": 500.3177185058594,
+      "completions/min_length": 234.0,
+      "completions/min_terminated_length": 234.0,
+      "entropy": 0.0772857666015625,
+      "epoch": 0.8018223234624146,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.029832610860466957,
+      "kl": 0.0002658963203430176,
+      "learning_rate": 3.625760102513103e-06,
+      "loss": -0.0039,
+      "num_tokens": 4544172.0,
+      "reward": 0.5655040740966797,
+      "reward_std": 0.10794761031866074,
+      "rewards/cross_solution_unittest/mean": 0.5655040740966797,
+      "rewards/cross_solution_unittest/std": 0.22446972131729126,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
       "step": 22
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 864.0,
+      "completions/max_terminated_length": 864.0,
+      "completions/mean_length": 509.30731201171875,
+      "completions/mean_terminated_length": 509.30731201171875,
       "completions/min_length": 243.0,
       "completions/min_terminated_length": 243.0,
+      "entropy": 0.0737457275390625,
+      "epoch": 0.8382687927107062,
       "frac_reward_zero_std": 0.0,
+      "grad_norm": 0.030700158327817917,
+      "kl": 0.0002843141555786133,
+      "learning_rate": 2.7103137257858867e-06,
+      "loss": 0.0074,
+      "num_tokens": 4752069.0,
+      "reward": 0.5879608392715454,
+      "reward_std": 0.11106596887111664,
+      "rewards/cross_solution_unittest/mean": 0.5879607796669006,
+      "rewards/cross_solution_unittest/std": 0.2123921662569046,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
+      "step": 23
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_mean": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
+      "completions/clipped_ratio": -4.9375,
       "completions/max_length": 3584.0,
+      "completions/max_terminated_length": 908.0,
+      "completions/mean_length": 567.7552490234375,
+      "completions/mean_terminated_length": 536.0053100585938,
+      "completions/min_length": 284.0,
+      "completions/min_terminated_length": 284.0,
+      "entropy": 0.069793701171875,
+      "epoch": 0.8747152619589977,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.028657682240009308,
+      "kl": 0.00025784969329833984,
+      "learning_rate": 1.9098300562505266e-06,
+      "loss": -0.0067,
+      "num_tokens": 4971092.0,
+      "reward": 0.5960028171539307,
+      "reward_std": 0.10218709707260132,
+      "rewards/cross_solution_unittest/mean": 0.5960028171539307,
+      "rewards/cross_solution_unittest/std": 0.19247692823410034,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
+      "step": 24
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 950.0,
+      "completions/max_terminated_length": 950.0,
+      "completions/mean_length": 494.65625,
+      "completions/mean_terminated_length": 494.65625,
+      "completions/min_length": 279.0,
+      "completions/min_terminated_length": 279.0,
+      "entropy": 0.078704833984375,
+      "epoch": 0.9111617312072893,
       "frac_reward_zero_std": 0.0416666679084301,
+      "grad_norm": 0.03116031549870968,
+      "kl": 0.00027370452880859375,
+      "learning_rate": 1.2369331995613664e-06,
+      "loss": 0.0021,
+      "num_tokens": 5166194.0,
+      "reward": 0.554139256477356,
+      "reward_std": 0.0889216810464859,
+      "rewards/cross_solution_unittest/mean": 0.554139256477356,
+      "rewards/cross_solution_unittest/std": 0.1982363760471344,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
+      "step": 25
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 767.0,
+      "completions/max_terminated_length": 767.0,
+      "completions/mean_length": 497.953125,
+      "completions/mean_terminated_length": 497.953125,
+      "completions/min_length": 262.0,
+      "completions/min_terminated_length": 262.0,
+      "entropy": 0.073150634765625,
+      "epoch": 0.9476082004555809,
       "frac_reward_zero_std": 0.02083333395421505,
+      "grad_norm": 0.029424885287880898,
+      "kl": 0.00023937225341796875,
+      "learning_rate": 7.022351411174866e-07,
+      "loss": -0.0041,
+      "num_tokens": 5368559.0,
+      "reward": 0.5750547647476196,
+      "reward_std": 0.11895763874053955,
+      "rewards/cross_solution_unittest/mean": 0.5750547647476196,
+      "rewards/cross_solution_unittest/std": 0.18430344760417938,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
+      "step": 26
     },
     {
       "clip_ratio/high_max": 0.0,
       "clip_ratio/low_min": 0.0,
       "clip_ratio/region_mean": 0.0,
       "completions/clipped_ratio": -5.0,
+      "completions/max_length": 986.0,
+      "completions/max_terminated_length": 986.0,
+      "completions/mean_length": 518.8073120117188,
+      "completions/mean_terminated_length": 518.8073120117188,
+      "completions/min_length": 270.0,
+      "completions/min_terminated_length": 270.0,
+      "entropy": 0.073272705078125,
+      "epoch": 0.9840546697038725,
+      "frac_reward_zero_std": 0.0,
+      "grad_norm": 0.030289024114608765,
+      "kl": 0.00026422739028930664,
+      "learning_rate": 3.1416838871368925e-07,
       "loss": 0.0135,
+      "num_tokens": 5576354.0,
+      "reward": 0.5601072311401367,
+      "reward_std": 0.10010793805122375,
+      "rewards/cross_solution_unittest/mean": 0.5601072311401367,
+      "rewards/cross_solution_unittest/std": 0.18709710240364075,
       "rewards/format_reward/mean": 0.0,
       "rewards/format_reward/std": 0.0,
+      "step": 27
     },
     {
+      "epoch": 0.9840546697038725,
+      "step": 27,
       "total_flos": 0.0,
+      "train_loss": 0.0014987390906618977,
+      "train_runtime": 3075.3471,
+      "train_samples_per_second": 0.428,
+      "train_steps_per_second": 0.009
     }
   ],
   "logging_steps": 1,
+  "max_steps": 28,
+  "num_input_tokens_seen": 5576354,
   "num_train_epochs": 1,
   "save_steps": 50,
   "stateful_callbacks": {

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0b58044669034d66de7eb7c75b2ade9e1a7cd8412ddc6e70e0864b5ae4d758e7
 size 8376

 version https://git-lfs.github.com/spec/v1
+oid sha256:85ae6de1dc95cad16f39f81abf09f813908d33a9096df6b4e8cffa07a28fb220
 size 8376