code-olmo2_1b_sft-nobuf
GRPO on MBPP from allenai/OLMo-2-0425-1B-SFT. Arm: GRPO, no replay buffer.
Part of the RL-forgetting code study: the same run is trained with and without an SFT-replay buffer and evaluated on MBPP+ across training. The sibling arm is RL-Forgettting-Exp4/code-olmo2_1b_sft-replay-hardcd-lam0p1. Training finished all 300 steps.
Layout
One subfolder per checkpoint, global_step_<N>/, each a full Hugging Face model directory (bf16 model.safetensors, config, tokenizer). Steps are 50โ300, stride 50. Optimizer, FSDP and replay-buffer state are not included. Steps 50โ250 are the in-training HF exports. The final step 300 only had FSDP shards, so it was merged offline from a copy of them with the same verl model merger code.
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "RL-Forgettting-Exp4/code-olmo2_1b_sft-nobuf"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="global_step_300", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="global_step_300")
Training
Source: jy-evangeline/rl-forgetting-experiments commit 1fa2cc9 (polaris-8gpu), reward-signal-analysis/script/rl_with_replay/code/olmo2_1b_sft_mbpp_nobuf.sbatch โ mbpp_nobuf.sh, experiment name olmo2_1b_sft_mbpp_nobuf. Run on the MSR Singularity cluster as jingyan-olmo2-mbpp-nobuf-20261117.
| setting | value |
|---|---|
| base model | allenai/OLMo-2-0425-1B-SFT (revision 0d85a3d037876ce6ac7d4311d994400fc66ac27f) |
| data | MBPP code-RL train split (320 prompts); native validation on the 378-prompt MBPP test split every 50 steps |
| algorithm | GRPO (verl), 8 rollouts per prompt, no advantage std normalization, no KL loss and no KL in the reward, entropy coefficient 0 (entropy logged) |
| reward | binary: execution of the generated program against the task's asserts, 10 s timeout |
| actor learning rate | 1e-6, constant (no warmup or decay); weight decay 0.01 |
| batch | 64 prompts ร 8 rollouts per step; PPO mini-batch 64 |
| lengths / sampling | prompt โค 1024, response โค 3072 tokens (fits OLMo-2's 4096-token context); rollout temperature 1.0 |
| replay buffer | none |
| steps | 300 (60 epochs), seed 1 |
| hardware | 2 ร H100 80GB |
Deviations from the sbatch
- Code-execution workers: 22, not 32. The 2รH100 cluster SKU gives the job a 22-CPU cgroup. The launcher sizes
CODE_MAX_WORKERSto the cgroup, because oversubscribed workers can time out and score correct programs 0. Before training, a preflight check scored all 320 train reference solutions correct with these workers. - Native checkpoint every 10 steps.
mbpp_nobuf.shhard-codesSAVE_FREQ=10, which overrides the sbatch's 50. As upstream, all but the last two checkpoints are converted to HF and their full state is pruned (KEEP_FULL_CKPT_LAST_N=2). This repo publishes every 50th step. - W&B offline. The cluster had no W&B credential, so W&B ran in offline mode under the same project and entity settings. The offline run directories are published with the training artifacts. Console logging is unchanged.
- Preemption. The job ran on preemptible capacity and was restarted 4 times, each time resuming automatically from the latest checkpoint (the cluster equivalent of the sbatch requeue). Updates after the last checkpoint were recomputed.
- Container: the patched November cluster image, with the same prebuilt Python runtime and the commit's own verl tree.
Held-out evaluation
MBPP+ (378 problems), 160 samples per problem (temperature 0.6, top-p 0.95, up to 3072 new tokens). Scoring uses the native execution scorer with MBPP+'s full test suite (~108 tests per problem), which is much stricter than MBPP's original 3 asserts. pass@k uses the unbiased estimator. Step 0 is the base model and is shared by both arms.
| step | source | pass@1 | pass@8 | pass@64 | pass@128 |
|---|---|---|---|---|---|
| 0 | base model | 0.1490 | 0.4106 | 0.6165 | 0.6648 |
| 50 | this run | 0.3385 | 0.4913 | 0.5863 | 0.6099 |
| 100 | this run | 0.3326 | 0.4268 | 0.4953 | 0.5107 |
| 150 | this run | 0.3330 | 0.4277 | 0.4823 | 0.4935 |
| 200 | this run | 0.3241 | 0.4019 | 0.4602 | 0.4730 |
| 250 | this run | 0.3251 | 0.4039 | 0.4484 | 0.4538 |
| 300 | this run | 0.3339 | 0.4058 | 0.4571 | 0.4662 |
Raw responses, per-question outcomes, pass@k for all k and learning/forgetting trajectories are in RL-Forgettting-Exp4/rebuttal-results-20261109 (latest snapshot evaluations/20261120-0810). Training rollouts, native validation, console metrics and the offline W&B directories are in mbpp-olmo2-1b-sft-nobuf/ in the same dataset.