JaeWooShin/peft-rlvr

PEFT for RLVR on Qwen3-4B-Base

GRPO runs with parameter-efficient adapters, following "Evaluating Parameter Efficient Methods for RLVR" (arXiv 2512.23165): LoRA, DoRA, PiSSA and MiLoRA, rank 32, alpha 64, dropout 0.05 on every linear layer of the decoder, learning rate 1e-5. Recipe: English DAPO-Math-17k, 16 answers per question, 64 questions per step, 4 optimizer updates per step, clip 0.2/0.28, no KL, 8k-token responses, 200 steps (verl 0.10, one GPU per run).

Validation every 25 steps (and before training): AIME24, AIME25, AMC23 (32 answers per problem) and MATH-500 (4 answers), at temperature 0.6 / top_p 0.95 (val-core/<benchmark>/acc/mean@4) and at temperature 1.0 / top_p 1.0 (val-core@t1.0/...), in each run's results/metrics.jsonl.

One folder per run (qwen3-4b-base_<method>_r32_lr1e-5_g16_b64_s0):

path	content
adapters/global_step_<N>/	the PEFT adapter every 25 steps (fp32)
adapters/init/	PiSSA, MiLoRA: the initial adapter A0, B0. These runs trained on W - s·B0·A0, so the model is W - s·B0·A0 + s·B·A
merged/global_step_200/	the final model with the adapter merged in (bf16 transformers)
checkpoint/global_step_200/	verl's LoRA-only checkpoint (adapter, optimizer, scheduler, data order)
rollouts/	the training rollouts, 25 steps per file
results/	metrics, validation outputs at both temperatures, TensorBoard, logs