--- license: apache-2.0 base_model: Qwen/Qwen2.5-1.5B-Instruct tags: - grpo - rlhf - deepseek-r1 - math-reasoning - gsm8k - vllm datasets: - openai/gsm8k metrics: - accuracy - reward --- # AetherControl-Qwen2.5-1.5B-GRPO-Math This model is fine-tuned using **Group Relative Policy Optimization (GRPO)** on 500 reasoning prompts from the GSM8K dataset as part of the **AetherControl** platform. --- ## ๐Ÿ“Š Training Telemetry Progression (20 Steps) ```text ๐Ÿ“Š GRPO Fine-Tuning Progression & Telemetry Log โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ณโ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”“ โ”ƒ Step โ”ƒ Mean Reward โ”ƒ Format โ”ƒ Accuracy โ”ƒ KL โ”ƒ GRPO Loss โ”ƒ โ”ƒ โ”ƒ (r_mean) โ”ƒ Reward โ”ƒ Reward โ”ƒ Div (D_KL) โ”ƒ (L_grpo) โ”ƒ โ”กโ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ•‡โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”โ”ฉ โ”‚ Step 01 โ”‚ 0.42 โ”‚ 0.23 โ”‚ 0.22 โ”‚ 0.0376 โ”‚ 1.1934 โ”‚ โ”‚ Step 05 โ”‚ 0.50 โ”‚ 0.34 โ”‚ 0.32 โ”‚ 0.0581 โ”‚ 0.9610 โ”‚ โ”‚ Step 10 โ”‚ 0.64 โ”‚ 0.54 โ”‚ 0.48 โ”‚ 0.0595 โ”‚ 0.7322 โ”‚ โ”‚ Step 15 โ”‚ 0.72 โ”‚ 0.75 โ”‚ 0.70 โ”‚ 0.0536 โ”‚ 0.4367 โ”‚ โ”‚ Step 20 โ”‚ 0.85 โ”‚ 0.92 โ”‚ 0.87 โ”‚ 0.0586 โ”‚ 0.1730 โ”‚ โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ ``` * **Mean Reward ($r_{\text{mean}}$):** Increased from **0.42 $\rightarrow$ 0.85** (+102% improvement). * **KL Divergence ($D_{\text{KL}}$):** Maintained under **0.0586** (stable policy). * **GRPO Loss ($\mathcal{L}_{\text{GRPO}}$):** Decreased from **1.1934 $\rightarrow$ 0.1730**. * **Verifier Rewards:** Rule-based format reward (`` tags) and exact math string match (`math_reward.py`). --- ## ๐Ÿ›๏ธ System Repository Github Repository: [https://github.com/Aravind0403/Building-and-Optimizing-Production-LLM-Serving-System](https://github.com/Aravind0403/Building-and-Optimizing-Production-LLM-Serving-System)