Conditional Length Penalty Eval Results
Eval Setup
- Model: Qwen/Qwen3-1.7B (base)
- Dataset: DeepScaleR, difficulty 0-50% (harder problems)
- Num problems: 100
- Num generations: 1 per problem
- Temperature: 0.6, top_p: 0.95
- Max tokens: 8192
- Correctness: math_verify symbolic equivalence
- Format check: has ... and \boxed{}
Results
| Checkpoint | Accuracy | Formatted | Mean Tokens | Median Tokens |
|---|---|---|---|---|
| base (Qwen3-1.7B) | 28.0% | 54.0% | 6,821 | 8,192 |
| cond lambda=1.0 step 50 | 15.0% | 100.0% | 730 | 450 |
| cond lambda=1.0 step 100 | 17.0% | 88.0% | 2,012 | 871 |
| cond lambda=1.0 step 150 | 18.0% | 96.0% | 2,174 | 1,240 |
| cond lambda=1.0 step 200 | 25.0% | 89.0% | 3,121 | 2,199 |
| cond lambda=0.3 step 50 | 16.0% | 76.0% | 837 | 534 |
| cond lambda=0.3 step 100 | 19.0% | 90.0% | 1,713 | 1,078 |
| cond lambda=0.3 step 150 | 3.0% | 27.0% | 2,774 | 2,049 |
| cond lambda=0.3 step 200 | 21.0% | 84.0% | 3,505 | 2,304 |
| cond lambda=0.05 step 50 | 17.0% | 97.0% | 1,242 | 908 |
| cond lambda=0.05 step 100 | 13.0% | 99.0% | 1,162 | 716 |
| cond lambda=0.05 step 150 | 5.0% | 38.0% | 2,855 | 1,580 |
Notes
- All checkpoints are conditional length penalty (penalty only applied when answer is correct)
- lambda=0.3 step 150 and lambda=0.05 step 150 both show severe accuracy collapse
- lambda=0.3 step 200 recovers to near-base accuracy (21% vs 28%) with much shorter responses
- lambda=0.05 was killed at step 177 so no step 200 checkpoint exists
- Eval on hard problems (0-50% solve rate) — training was on easier problems (50-95%)