---
license: apache-2.0
base_model: Qwen/Qwen3-8B-Base
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- math
- reinforcement-learning
- rlsd
- verl
---

# Qwen3-8B-Base RLSD

RLSD self-distillation reinforcement learning on the local math training split.

This repository contains the final merged Hugging Face checkpoint from `global_step_100`.
The training checkpoint was saved from FSDP shards and merged to safetensors for this upload.

## Training Method

- Policy loss mode: `rlsd`.
- Self-distillation uses reprompt feedback and token reweighting.
- Token reweighting: lambda 0.5, eps_w 0.2, decay steps 50.
- Advantage estimator remains `grpo` in the trainer config.
- Reward function: local math `compute_score` reward manager.
- Fine-tuning type: full-parameter FSDP training, not LoRA.

## Training Hyperparameters

| Field | Value |
|---|---:|
| Base model | `Qwen/Qwen3-8B-Base` |
| Train file | `/home1/irteam/SDPO/self-distillation-analysis/data/math/train.parquet` |
| Validation file | `/home1/irteam/SDPO/self-distillation-analysis/data/math/evaluation/aime24.parquet` |
| Train max samples | 25600 |
| Train batch size | 256 |
| Rollouts per prompt | 8 |
| PPO mini batch size | 128 |
| PPO micro batch size per GPU | 1 |
| Optimizer | AdamW |
| Learning rate | 1e-06 |
| Weight decay | 0.01 |
| LR warmup steps | 10 |
| Total training steps | 100 |
| Save frequency | every 10 steps |
| Validation frequency | every 10 steps |
| Max prompt length | 2048 |
| Max response length | 20480 |
| Rollout backend | vllm |
| Rollout temperature | 1 |
| Rollout top_p | 1 |
| vLLM GPU memory utilization | 0.75 |
| Actor strategy | fsdp |
| Dtype | bfloat16 |
| Advantage estimator | grpo |
| Gamma / Lambda | 1 / 1 |
| KL loss enabled | False |
| KL loss coefficient | 0.001 |
| Checkpoint uploaded | `math-RLSD-Qwen3-8B-Base-128-train256-rollout8-lr1e-6-vllm0.75-modelQwen-Qwen3-8B-Base/global_step_100` |
| W&B run id | `z0qjc5ge` |

## Training Score

The plot below shows `critic/score/mean` logged during training.

![Training score](training_score.png)

CSV data is included in [`training_score.csv`](training_score.csv).

| Metric | Value |
|---|---:|
| Final training step | 100 |
| Final `critic/score/mean` | 0.468262 |
| Final `val-core/math_dapo/acc/mean@1` | 0.266667 |

## Intended Use

This model is intended for internal research and analysis of math-focused RL fine-tuning methods.
It has not been broadly safety evaluated for production use.

## Limitations

The model was trained for 100 optimization steps on a local math dataset split.
Reported scores are training-time reward/validation metrics from the same experiment setup and should not be treated as broad benchmark results.