OpenWebRL-4B: pooled-data vs split-schedule RPO (LoRA adapters)

Two LoRA adapters (r 16, alpha 32, all attention + MLP projections) trained from OpenWebRL-4B-SFT with Round 1's RPO recipe (DPO beta 0.1 + SFT term weight 1.0 on successful-episode pairs, reference = OpenWebRL-4B-SFT, lr peak 5e-5, warmup 512 examples, half cosine to 2.5e-5, batch 32, 2 H200). Both see the same 11,554 Sol pairs (half A = Round 1 data, 5,304 pairs; half B = data collected by the Round 1 model, 6,250 pairs) twice: 723 updates.

folder experiment data order
pooled/ Experiment 1: accumulate both rounds' data, train once from SFT A + B shuffled together, 2 epochs
seqsched/ Experiment 2: two rounds as ONE LR schedule (no re-warmup) 2 epochs of A, then 2 epochs of B; LR 4.41e-5 at the switch (update 332)

Each folder: update-000723/ = final adapter; earlier checkpoints update-000181/362/542 (seqsched/ also update-000332, the end of half A); recipe/ = frozen config, metrics, held-out validation, the training sbatch.

Load: PeftModel.from_pretrained(<OpenWebRL-4B-SFT>, "<repo>", subfolder="pooled/update-000723"). adapter_config.json names the base by its local cluster path; point it at your copy of OpenWebRL-4B-SFT. Not yet evaluated.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support