Add model card and training score plot

README.md

@@ -0,0 +1,86 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-4B-Base
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- qwen3
+- math
+- reinforcement-learning
+- rlsd
+- verl
+---
+
+# Qwen3-4B-Base RLSD
+
+RLSD self-distillation reinforcement learning on the local math training split.
+
+This repository contains the final merged Hugging Face checkpoint from `global_step_100`.
+The training checkpoint was saved from FSDP shards and merged to safetensors for this upload.
+
+## Training Method
+
+- Policy loss mode: `rlsd`.
+- Self-distillation uses reprompt feedback and token reweighting.
+- Token reweighting: lambda 0.5, eps_w 0.2, decay steps 50.
+- Advantage estimator remains `grpo` in the trainer config.
+- Reward function: local math `compute_score` reward manager.
+- Fine-tuning type: full-parameter FSDP training, not LoRA.
+
+## Training Hyperparameters
+
+| Field | Value |
+|---|---:|
+| Base model | `Qwen/Qwen3-4B-Base` |
+| Train file | `/home1/irteam/SDPO/self-distillation-analysis/data/math/train.parquet` |
+| Validation file | `/home1/irteam/SDPO/self-distillation-analysis/data/math/evaluation/aime24.parquet` |
+| Train max samples | 25600 |
+| Train batch size | 256 |
+| Rollouts per prompt | 8 |
+| PPO mini batch size | 128 |
+| PPO micro batch size per GPU | 1 |
+| Optimizer | AdamW |
+| Learning rate | 1e-06 |
+| Weight decay | 0.01 |
+| LR warmup steps | 10 |
+| Total training steps | 100 |
+| Save frequency | every 10 steps |
+| Validation frequency | every 10 steps |
+| Max prompt length | 2048 |
+| Max response length | 20480 |
+| Rollout backend | vllm |
+| Rollout temperature | 1 |
+| Rollout top_p | 1 |
+| vLLM GPU memory utilization | 0.75 |
+| Actor strategy | fsdp |
+| Dtype | bfloat16 |
+| Advantage estimator | grpo |
+| Gamma / Lambda | 1 / 1 |
+| KL loss enabled | False |
+| KL loss coefficient | 0.001 |
+| Checkpoint uploaded | `math-RLSD-Qwen3-4B-Base-128-train256-rollout8-lr1e-6-vllm0.75-modelQwen-Qwen3-4B-Base/global_step_100` |
+| W&B run id | `3tuehy90` |
+
+## Training Score
+
+The plot below shows `critic/score/mean` logged during training.
+
+![Training score](training_score.png)
+
+CSV data is included in [`training_score.csv`](training_score.csv).
+
+| Metric | Value |
+|---|---:|
+| Final training step | 100 |
+| Final `critic/score/mean` | 0.304199 |
+| Final `val-core/math_dapo/acc/mean@1` | 0.1 |
+
+## Intended Use
+
+This model is intended for internal research and analysis of math-focused RL fine-tuning methods.
+It has not been broadly safety evaluated for production use.
+
+## Limitations
+
+The model was trained for 100 optimization steps on a local math dataset split.
+Reported scores are training-time reward/validation metrics from the same experiment setup and should not be treated as broad benchmark results.

training_score.csv

@@ -0,0 +1,101 @@
+step,critic_score_mean,val_math_dapo_acc_mean_at_1
+1,0.1162109375,
+2,0.1142578125,
+3,0.12109375,
+4,0.11328125,
+5,0.1298828125,
+6,0.13525390625,
+7,0.11376953125,
+8,0.13720703125,
+9,0.138671875,
+10,0.14013671875,0.13333333333333333
+11,0.125,
+12,0.15576171875,
+13,0.1279296875,
+14,0.15185546875,
+15,0.1591796875,
+16,0.16796875,
+17,0.1416015625,
+18,0.16552734375,
+19,0.1708984375,
+20,0.1767578125,0.1
+21,0.1787109375,
+22,0.1826171875,
+23,0.22265625,
+24,0.1875,
+25,0.17236328125,
+26,0.23388671875,
+27,0.22216796875,
+28,0.20166015625,
+29,0.26416015625,
+30,0.2275390625,0.13333333333333333
+31,0.25537109375,
+32,0.22705078125,
+33,0.23095703125,
+34,0.24267578125,
+35,0.232421875,
+36,0.24365234375,
+37,0.2421875,
+38,0.21728515625,
+39,0.2431640625,
+40,0.2626953125,0.1
+41,0.2666015625,
+42,0.25634765625,
+43,0.26904296875,
+44,0.2685546875,
+45,0.25341796875,
+46,0.275390625,
+47,0.291015625,
+48,0.26416015625,
+49,0.29345703125,
+50,0.25,0.13333333333333333
+51,0.28515625,
+52,0.2470703125,
+53,0.25390625,
+54,0.31689453125,
+55,0.25048828125,
+56,0.3115234375,
+57,0.25927734375,
+58,0.29345703125,
+59,0.271484375,
+60,0.25732421875,0.16666666666666666
+61,0.30224609375,
+62,0.2783203125,
+63,0.32177734375,
+64,0.2568359375,
+65,0.2900390625,
+66,0.26904296875,
+67,0.265625,
+68,0.2998046875,
+69,0.236328125,
+70,0.27978515625,0.13333333333333333
+71,0.30419921875,
+72,0.27294921875,
+73,0.302734375,
+74,0.27880859375,
+75,0.28466796875,
+76,0.3046875,
+77,0.306640625,
+78,0.31689453125,
+79,0.30029296875,
+80,0.3173828125,0.1
+81,0.29638671875,
+82,0.3330078125,
+83,0.33837890625,
+84,0.287109375,
+85,0.28955078125,
+86,0.30810546875,
+87,0.326171875,
+88,0.3515625,
+89,0.3203125,
+90,0.3046875,0.16666666666666666
+91,0.322265625,
+92,0.33056640625,
+93,0.30224609375,
+94,0.34521484375,
+95,0.33984375,
+96,0.30322265625,
+97,0.31640625,
+98,0.32373046875,
+99,0.3447265625,
+100,0.30419921875,0.1