code-olmo2_1b_sft-nobuf

GRPO on MBPP from allenai/OLMo-2-0425-1B-SFT. Arm: GRPO, no replay buffer.

Part of the RL-forgetting code study: the same run is trained with and without an SFT-replay buffer and evaluated on MBPP+ across training. The sibling arm is RL-Forgettting-Exp4/code-olmo2_1b_sft-replay-hardcd-lam0p1. Training finished all 300 steps.

Layout

One subfolder per checkpoint, global_step_<N>/, each a full Hugging Face model directory (bf16 model.safetensors, config, tokenizer). Steps are 50โ€“300, stride 50. Optimizer, FSDP and replay-buffer state are not included. Steps 50โ€“250 are the in-training HF exports. The final step 300 only had FSDP shards, so it was merged offline from a copy of them with the same verl model merger code.

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "RL-Forgettting-Exp4/code-olmo2_1b_sft-nobuf"
model = AutoModelForCausalLM.from_pretrained(repo, subfolder="global_step_300", torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder="global_step_300")

Training

Source: jy-evangeline/rl-forgetting-experiments commit 1fa2cc9 (polaris-8gpu), reward-signal-analysis/script/rl_with_replay/code/olmo2_1b_sft_mbpp_nobuf.sbatch โ†’ mbpp_nobuf.sh, experiment name olmo2_1b_sft_mbpp_nobuf. Run on the MSR Singularity cluster as jingyan-olmo2-mbpp-nobuf-20261117.

setting value
base model allenai/OLMo-2-0425-1B-SFT (revision 0d85a3d037876ce6ac7d4311d994400fc66ac27f)
data MBPP code-RL train split (320 prompts); native validation on the 378-prompt MBPP test split every 50 steps
algorithm GRPO (verl), 8 rollouts per prompt, no advantage std normalization, no KL loss and no KL in the reward, entropy coefficient 0 (entropy logged)
reward binary: execution of the generated program against the task's asserts, 10 s timeout
actor learning rate 1e-6, constant (no warmup or decay); weight decay 0.01
batch 64 prompts ร— 8 rollouts per step; PPO mini-batch 64
lengths / sampling prompt โ‰ค 1024, response โ‰ค 3072 tokens (fits OLMo-2's 4096-token context); rollout temperature 1.0
replay buffer none
steps 300 (60 epochs), seed 1
hardware 2 ร— H100 80GB

Deviations from the sbatch

  • Code-execution workers: 22, not 32. The 2ร—H100 cluster SKU gives the job a 22-CPU cgroup. The launcher sizes CODE_MAX_WORKERS to the cgroup, because oversubscribed workers can time out and score correct programs 0. Before training, a preflight check scored all 320 train reference solutions correct with these workers.
  • Native checkpoint every 10 steps. mbpp_nobuf.sh hard-codes SAVE_FREQ=10, which overrides the sbatch's 50. As upstream, all but the last two checkpoints are converted to HF and their full state is pruned (KEEP_FULL_CKPT_LAST_N=2). This repo publishes every 50th step.
  • W&B offline. The cluster had no W&B credential, so W&B ran in offline mode under the same project and entity settings. The offline run directories are published with the training artifacts. Console logging is unchanged.
  • Preemption. The job ran on preemptible capacity and was restarted 4 times, each time resuming automatically from the latest checkpoint (the cluster equivalent of the sbatch requeue). Updates after the last checkpoint were recomputed.
  • Container: the patched November cluster image, with the same prebuilt Python runtime and the commit's own verl tree.

Held-out evaluation

MBPP+ (378 problems), 160 samples per problem (temperature 0.6, top-p 0.95, up to 3072 new tokens). Scoring uses the native execution scorer with MBPP+'s full test suite (~108 tests per problem), which is much stricter than MBPP's original 3 asserts. pass@k uses the unbiased estimator. Step 0 is the base model and is shared by both arms.

step source pass@1 pass@8 pass@64 pass@128
0 base model 0.1490 0.4106 0.6165 0.6648
50 this run 0.3385 0.4913 0.5863 0.6099
100 this run 0.3326 0.4268 0.4953 0.5107
150 this run 0.3330 0.4277 0.4823 0.4935
200 this run 0.3241 0.4019 0.4602 0.4730
250 this run 0.3251 0.4039 0.4484 0.4538
300 this run 0.3339 0.4058 0.4571 0.4662

Raw responses, per-question outcomes, pass@k for all k and learning/forgetting trajectories are in RL-Forgettting-Exp4/rebuttal-results-20261109 (latest snapshot evaluations/20261120-0810). Training rollouts, native validation, console metrics and the offline W&B directories are in mbpp-olmo2-1b-sft-nobuf/ in the same dataset.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Model tree for RL-Forgettting-Exp4/code-olmo2_1b_sft-nobuf

Finetuned
(5)
this model