Upload DGPO-Training/DGPO-Training-README.md with huggingface_hub

DGPO-Training/DGPO-Training-README.md

@@ -0,0 +1,119 @@
+# Qwen3-0.6B DGPO Training
+
+**Difficulty-Aware Group Policy Optimization (DGPO)** - ICLR 2026
+
+## What is DGPO?
+
+DGPO is an advanced reinforcement learning method that extends GRPO (Group Relative Policy Optimization) by adding difficulty-aware mechanisms. It's from the **MathForge** paper: "Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO".
+
+### Key Innovation
+
+Standard GRPO has an implicit imbalance where **harder questions get lower policy updates**. DGPO fixes this by:
+
+1. **Difficulty-Balanced Group Advantage Estimation**
+   - Uses Mean Absolute Deviation (MAD) instead of standard deviation
+   - Normalizes advantages based on question difficulty
+
+2. **Difficulty-Aware Question Weighting (DQW)**
+   - Prioritizes harder questions during training
+   - Uses softmax weighting with temperature parameter
+
+## DGPO vs DPO vs GRPO
+
+| Method | Type | Key Feature | Best For |
+|--------|------|-------------|----------|
+| **DPO** | Preference | Pairwise preferences | General alignment |
+| **GRPO** | RLVR | Group-based rewards | Math reasoning |
+| **DGPO** | RLVR | Difficulty-aware | Hard math problems |
+
+## Installation
+
+```bash
+# Install dependencies
+pip install torch transformers trl peft datasets accelerate
+
+# Clone MathForge (optional, for reference)
+git clone https://github.com/AMAP-ML/MathForge.git
+```
+
+## Quick Start
+
+### Basic DGPO Training
+
+```bash
+python train_dgpo_qwen3.py \
+    --model_name Qwen/Qwen3-0.6B \
+    --enable_dgpo \
+    --enable_dgpo_dqw \
+    --dgpo_dqw_temp 2.0
+```
+
+### With Custom Parameters
+
+```bash
+python train_dgpo_qwen3.py \
+    --model_name Qwen/Qwen3-0.6B \
+    --dataset_name DigitalLearningGmbH/MATH-lighteval \
+    --enable_dgpo \
+    --enable_dgpo_dqw \
+    --dgpo_dqw_temp 2.0 \
+    --num_generations 8 \
+    --max_completion_length 1024 \
+    --learning_rate 1e-6 \
+    --num_train_epochs 3
+```
+
+### Using Shell Script
+
+```bash
+chmod +x run_dgpo.sh
+./run_dgpo.sh
+```
+
+## DGPO Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--enable_dgpo` | True | Enable DGPO algorithm |
+| `--enable_dgpo_dqw` | True | Enable Difficulty-aware Question Weighting |
+| `--dgpo_dqw_temp` | 2.0 | Temperature for DQW (higher = more focus on hard questions) |
+
+## Training Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--num_generations` | 4 | Number of completions per prompt |
+| `--max_completion_length` | 512 | Maximum tokens in completion |
+| `--learning_rate` | 5e-7 | Learning rate (use small LR for DGPO) |
+| `--num_train_epochs` | 1 | Number of training epochs |
+| `--beta` | 0.0 | KL coefficient (0 = no reference model) |
+| `--temperature` | 0.7 | Sampling temperature |
+
+## Performance
+
+From the MathForge paper (Qwen2.5-Math-7B):
+
+| Method | AIME24 | AIME25 | AMC23 | MATH500 | Avg. |
+|--------|--------|--------|-------|---------|------|
+| GRPO | 20.94 | 8.44 | 58.98 | 72.20 | 37.61 |
+| **DGPO** | 23.85 | 10.21 | **61.02** | 74.25 | **39.79** |
+| MathForge | 24.58 | **12.60** | 59.84 | **79.95** | **42.17** |
+
+**DGPO improves over GRPO by +2.18% on average!**
+
+## References
+
+- **Paper**: [Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO](https://arxiv.org/abs/2601.20614)
+- **Code**: [AMAP-ML/MathForge](https://github.com/AMAP-ML/MathForge)
+- **Conference**: ICLR 2026
+
+## Citation
+
+```bibtex
+@article{dai2026harder,
+  title={Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation},
+  author={Dai, Yanqi and Ji, Yuxiang and Zhang, Xiao and Wang, Yong and Chu, Xiangxiang and Lu, Zhiwu},
+  journal={ICLR},
+  year={2026}
+}
+```