README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-4B
tags:
  - reinforcement-learning
  - grpo
  - gdpo
  - math
  - lora
  - qwen3
datasets:
  - agentica-org/DeepScaleR-Preview-Dataset
---

# DeepScaleR GDPO Length Penalty - LoRA Adapters

LoRA adapters from GRPO training with GDPO (Group Direct Preference Optimization) length penalty on the DeepScaleR math reasoning dataset.

## Training Overview

**Objective**: Train a model to solve math problems correctly while penalizing unnecessarily long responses, using a combined reward signal with length penalty.

**Reward Function (Sum-then-Normalize GDPO)**:
For each group of rollouts from the same prompt:
1. Compute raw combined score: `correctness + lambda * length_penalty`
   - `correctness`: 1.0 if answer matches ground truth, 0.0 otherwise
   - `length_penalty`: `-num_tokens / max_completion_length`
2. Z-normalize the combined scores within each group (zero mean, unit variance)
3. Use normalized scores as advantages for GRPO policy gradient update

This "sum-then-normalize" approach ensures the length penalty signal is proportional to lambda relative to correctness before normalization, rather than independently normalizing each component.

## Training Configuration

| Parameter | Value |
|---|---|
| **Base Model** | Qwen/Qwen3-4B |
| **Method** | GRPO with GDPO length penalty |
| **Lambda (length penalty weight)** | 0.5 |
| **LoRA rank** | 16 |
| **LoRA alpha** | 16 |
| **LoRA target modules** | all-linear (q, k, v, o, gate, up, down proj) |
| **Learning rate** | 5e-5 (constant schedule) |
| **KL penalty (beta)** | 0.0 |
| **Temperature** | 1.0 |
| **Top-p** | 0.95 |
| **Prompts per step** | 6 |
| **Rollouts per prompt** | 8 |
| **Batch size** | 48 (6 x 8) |
| **Max completion length** | 15,000 tokens |
| **Total steps** | 200 |
| **Optimizer** | AdamW (weight_decay=0.1, betas=[0.9, 0.99]) |
| **Max grad norm** | 1.0 |
| **PPO epochs** | 1 |
| **Clip ratio** | 0.2 |
| **Framework** | Verl v0.6.1 + FSDP2 |
| **Hardware** | 4x NVIDIA H200 |

## Dataset

- **Source**: DeepScaleR math reasoning dataset (`agentica-org/DeepScaleR-Preview-Dataset`)
- **Difficulty filter**: 99-100% solved percentage (easiest problems)
- **Size**: 1,840 problems after filtering
- **System prompt**: "Please reason through this problem for yourself, and then when giving your answer ONLY provide your final answer within \boxed{} and DO NOT provide any reasoning."

## Key Training Metrics

| Metric | Step 1 | Step 100 | Step 199 |
|---|---|---|---|
| Mean response length | 1,414 | 61 | 92 |
| Max response length | 4,289 | 134 | 204 |
| KL divergence | 0.0 | 0.29 | 0.22 |
| Policy loss | 0.032 | 0.003 | 0.007 |
| Grad norm | 0.019 | 0.195 | 0.149 |
| Entropy | 0.140 | 0.057 | 0.061 |

The model learned to produce dramatically shorter responses (from ~1,400 tokens to ~60-90 tokens average) while maintaining correctness on easy math problems.

## Checkpoints

LoRA adapters are saved every 20 steps:
- `checkpoints/global_step_20/` through `checkpoints/global_step_200/`

Each checkpoint contains:
- `adapter_model.safetensors` (~127MB) - LoRA adapter weights
- `adapter_config.json` - PEFT/LoRA configuration

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")

# Load a specific checkpoint
model = PeftModel.from_pretrained(base_model, "brikdavies/deepscaleR/checkpoints/global_step_200")
```

## Run Details

- **Run ID**: `20260223_182638_deepscaler_gdpo_lambda0.5`
- **Training time**: ~71 minutes (200 steps)
- **Wandb project**: `grpo-deepscaler`