Conditional Length Penalty Lambda Sweep
Experiment Description
GRPO training on DeepScaleR math problems with conditional length penalty. Conditional mode: length penalty only applied when the model gets the answer correct. Reward = correctness - lambda * (tokens / max_tokens) if correct, else 0.0
Lambda Values
- lambda=1.0 (steps 50, 100, 150, 200)
- lambda=0.3 (steps 50, 100 — still running)
- lambda=0.1 (queued)
- lambda=0.05 (running)
Common Config
| Parameter | Value |
|---|---|
| Model | Qwen/Qwen3-1.7B |
| KL beta | 0.01 |
| Learning rate | 5e-5 |
| LR scheduler | constant |
| Warmup steps | 10 |
| Max steps | 200 |
| Max completion tokens | 15,000 |
| Max prompt tokens | 1,024 |
| Prompts per step | 8 |
| Rollouts per prompt | 8 |
| per_device_batch_size | 8 |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| Seed | 42 |
| Temperature | 1.0 |
| top_p | 0.95 |
| enable_thinking | True |
| gdpo_mode | False |
| norm_adv_by_std_in_grpo | True |
| Save frequency | Every 50 steps |
| Optimizer | AdamW 8-bit |
| Weight decay | 0.1 |
| Adam betas | (0.9, 0.99) |
| Max grad norm | 1.0 |
| KL loss type | low_var_kl |
| Gradient checkpointing | True |
System Prompt
Please reason through this problem for yourself, and then when giving your answer ONLY provide your final answer within \boxed{} and DO NOT provide any reasoning.
system_prompt_method: replace
Dataset
DeepScaleR (lime-nlp/deepscaleR_difficulty) filtered to solved_percentage 50-95%, max 1024 prompt tokens. 12,789 problems.
Reward Function
MathLengthPenaltyReward with:
- correct_reward: 1.0
- max_answer_tokens: 0 (unlimited)
- max_completion_length: 15,000
- conditional: True
- lambda_length: varies per run
WandB
Project: length_penalty_experiments
Checkpoint Format
LoRA adapters only (no full model weights). Base model: Qwen/Qwen3-1.7B. Load with: peft.PeftModel.from_pretrained(base_model, adapter_path)