conditional_lambda_sweep/config.md

# Conditional Length Penalty Lambda Sweep

## Experiment Description

GRPO training on DeepScaleR math problems with conditional length penalty.
Conditional mode: length penalty only applied when the model gets the answer correct.
Reward = correctness - lambda * (tokens / max_tokens) if correct, else 0.0

## Lambda Values

- lambda=1.0 (steps 50, 100, 150, 200)
- lambda=0.3 (steps 50, 100 — still running)
- lambda=0.1 (queued)
- lambda=0.05 (running)

## Common Config

| Parameter | Value |
|-----------|-------|
| Model | Qwen/Qwen3-1.7B |
| KL beta | 0.01 |
| Learning rate | 5e-5 |
| LR scheduler | constant |
| Warmup steps | 10 |
| Max steps | 200 |
| Max completion tokens | 15,000 |
| Max prompt tokens | 1,024 |
| Prompts per step | 8 |
| Rollouts per prompt | 8 |
| per_device_batch_size | 8 |
| LoRA rank | 32 |
| LoRA alpha | 64 |
| Seed | 42 |
| Temperature | 1.0 |
| top_p | 0.95 |
| enable_thinking | True |
| gdpo_mode | False |
| norm_adv_by_std_in_grpo | True |
| Save frequency | Every 50 steps |
| Optimizer | AdamW 8-bit |
| Weight decay | 0.1 |
| Adam betas | (0.9, 0.99) |
| Max grad norm | 1.0 |
| KL loss type | low_var_kl |
| Gradient checkpointing | True |

## System Prompt

```
Please reason through this problem for yourself, and then when giving your answer ONLY provide your final answer within \boxed{} and DO NOT provide any reasoning.
```

system_prompt_method: replace

## Dataset

DeepScaleR (lime-nlp/deepscaleR_difficulty) filtered to solved_percentage 50-95%, max 1024 prompt tokens. 12,789 problems.

## Reward Function

MathLengthPenaltyReward with:
- correct_reward: 1.0
- max_answer_tokens: 0 (unlimited)
- max_completion_length: 15,000
- conditional: True
- lambda_length: varies per run

## WandB

Project: length_penalty_experiments

## Checkpoint Format

LoRA adapters only (no full model weights). Base model: Qwen/Qwen3-1.7B.
Load with: peft.PeftModel.from_pretrained(base_model, adapter_path)