Add rlmath_30b_mlxp_v2, rlmath_q35_4b_kt_v1, rlmath_q35_4b_jupiter_v1 checkpoints

README.md

@@ -10,6 +10,7 @@ tags:
 base_model:
 - Qwen/Qwen3-4B-Thinking-2507
 - Qwen/Qwen3-30B-A3B-Thinking-2507
+- Qwen/Qwen3.5-4B
 ---
 
 # rlmath agentic GRPO checkpoints
@@ -20,14 +21,18 @@ Each folder is a standalone Hugging Face model directory with safetensors weight
 
 ## Common setup
 
-- **Trainer:** verl (fully-async policy, partial rollout, staleness 0.5) with the Alibaba agentic recipe (remote agent loop and LLM proxy). vLLM rollouts, FSDP training.
-- **Algorithm:** GRPO, lr 1e-6, clip 0.2 / 0.28, no KL loss, token-mean loss aggregation, temperature 1.0.
+- **Trainer:** verl (fully-async policy, partial rollout) with the Alibaba agentic recipe (remote agent loop and LLM proxy). vLLM rollouts, FSDP training.
+- **Algorithm:** GRPO, lr 1e-6 (constant), clip 0.2 / 0.28, no KL loss, token-mean loss aggregation, temperature 1.0, one optimizer step per training step.
 - **Reward:** construct tasks give 1/0. Optimize tasks give 0 if the submission is invalid, otherwise 0.1 + 0.9 · clipped progress toward the target.
-- **Validation:** 139 tasks, one per family, temperature 0.6, top-p 0.95. **Caveat:** that set was sampled from the training pool, and 93 of the 139 tasks are also in the 1,145-task training set used from `rlmath_async_v2` onward. The validation numbers below therefore partly measure training tasks, not held-out generalization.
+- **Validation:** one sample per task at temperature 0.6, top-p 0.95. Two validation sets were used:
+  - **v1 (contaminated):** 139 tasks, one per family. They were sampled from the training pool, and 93 of the 139 are also in the 1,145-task training set used from `rlmath_async_v2` onward. These numbers partly measure training tasks.
+  - **v2 (held out):** 121 tasks across 48 families with **no overlap** with the 1,024-task training set `tasks_train_v2`. All runs from the "v2 recipe" section below use it.
 
 ## Checkpoints
 
-| folder | base | run settings | validation (contaminated, see above) |
+### v1 recipe (validation set v1, contaminated)
+
+| folder | base | run settings | validation |
 |---|---|---|---|
 | `qwen3-4b-thinking/rlmath_q4bthink_async_v1/step_25` | Qwen3-4B-Thinking-2507 | 16 prompts × 8 rollouts, 8 turns, 8k tokens/call, 32k total, 3,401 tasks | step 25: 0.235 |
 | `qwen3-4b-thinking/rlmath_q4bthink_async_v1/step_50` | 〃 | 〃 | — |
@@ -36,8 +41,22 @@ Each folder is a standalone Hugging Face model directory with safetensors weight
 | `qwen3-30b-a3b-thinking/rlmath_30b_v1/step_6` | Qwen3-30B-A3B-Thinking-2507 | 16 × 8, 8 turns, 12k/call, 32k total, 1,145 tasks | step 0: 0.341 |
 | `qwen3-30b-a3b-thinking/rlmath_30b_mlxp_v1/step_36`, `step_48` | Qwen3-30B-A3B-Thinking-2507 | 16 × 32, 12 turns, 16k/call, 64k total, 1,145 tasks | 0.331 / 0.372 / 0.447 / 0.521 / 0.503 at steps 0 / 12 / 24 / 36 / 48 |
 
+### v2 recipe (validation set v2, held out)
+
+Shared settings: 1,024 training tasks, 16 prompts × 32 rollouts per step, up to 12 turns, **32k tokens per model call, 128k tokens per trajectory**, 192 planned steps (3 passes). Trajectories cut off by a length limit are dropped from the loss, and generation stops once a trajectory's budget is spent. Token-level truncated importance sampling (cap 2.0) corrects for the vLLM-vs-trainer mismatch, and each turn after the first is prompted with exactly the token sequence the trainer trains on. Weights are bf16 with torchao AdamW (bf16 stochastic rounding) under FSDP1.
+
+| folder | base | run-specific settings | held-out validation (v2) |
+|---|---|---|---|
+| `qwen3-30b-a3b-thinking/rlmath_30b_mlxp_v2/step_36`, `step_48` | Qwen3-30B-A3B-Thinking-2507 | 4 vLLM + 4 trainer GPUs, Ulysses SP 2, staleness 0.5, entropy bonus 0.001 | 0.427 / 0.479 / 0.548 / **0.536** / **0.539** at steps 0 / 12 / 24 / 36 / 48 |
+| `qwen3.5-4b/rlmath_q35_4b_kt_v1/step_24`, `step_36` | Qwen3.5-4B | 5 vLLM + 2 trainer GPUs, staleness 1.0, **entropy bonus 0** | 0.399 / 0.323 / **0.483** / **0.529** at steps 0 / 12 / 24 / 36 |
+| `qwen3.5-4b/rlmath_q35_4b_jupiter_v1/step_6`, `step_12`, `step_14`, `step_18`, `step_20` | Qwen3.5-4B | one 4×GH200 node (2 vLLM + 2 trainer GPUs), staleness 1.0, entropy bonus 0.001 | 0.343 / **0.511** at steps 0 / 12 (validated every 12 steps only) |
+
+**Qwen3.5 notes.** Qwen3.5 is a hybrid Gated-DeltaNet / attention model and is natively multimodal. Training was text-only: the vision tower is included in these folders but was never updated. Micro-batches hold one sequence each, because transformers' Gated-DeltaNet does not respect packed-sequence boundaries.
+
+**Entropy blowup in `rlmath_q35_4b_jupiter_v1`.** With the 0.001 entropy bonus, token entropy rose monotonically from about 0.66 (step 6) to 0.91 (step 12), 1.16 (step 14), 1.49 (step 18) and 1.82 (step 20), while responses grew to 30–51k tokens. A Qwen3.5-9B run with the same bonus collapsed after the same pattern (entropy 3.6, train reward halved by step 32). **Step 12 is the last checkpoint before the blowup**; steps 14–20 are included for analysis and are not recommended for use. The KT run of the same model uses no entropy bonus and stayed stable.
+
 ## Known issue in the earlier 4B runs
 
-`rlmath_q4bthink_async_v1` and `rlmath_async_v2` were trained with a train/rollout context mismatch. Qwen3-Thinking's chat template drops the reasoning of earlier assistant turns, so turns after the first were sampled without that reasoning in context but trained with it (vLLM-vs-trainer KL ≈ 0.045). Both runs degraded after about 25 steps. All later runs (`rlmath_4b_kt_v1`, `rlmath_30b_*`) prompt every turn with exactly the token sequence the trainer sees, which brings the KL down to about 1e-3.
+`rlmath_q4bthink_async_v1` and `rlmath_async_v2` were trained with a train/rollout context mismatch. Qwen3-Thinking's chat template drops the reasoning of earlier assistant turns, so turns after the first were sampled without that reasoning in context but trained with it (vLLM-vs-trainer KL ≈ 0.045). Both runs degraded after about 25 steps. All later runs (`rlmath_4b_kt_v1`, `rlmath_30b_*`, `rlmath_q35_*`) prompt every turn with exactly the token sequence the trainer sees, which brings the KL down to about 1e-3.
 
 The 30B runs use bf16 weights with torchao's AdamW (bf16 stochastic rounding) under FSDP1.