caiyuchen
/

DAPO-step-2

@@ -17,11 +17,20 @@ base_model:
 # On Predictability of Reinforcement Learning Dynamics for Large Language Models
 This repository provides one of the models used in our paper **"On Predictability of Reinforcement Learning Dynamics for Large Language Models"** for evaluating and predicting reinforcement learning (RL) dynamics in large language models (LLMs).
 Recent advances in LLM reasoning capabilities are largely driven by RL, yet the parameter dynamics during RL training remain poorly understood. Our work identifies two key properties of RL-induced parameter updates: **Rank-1 Dominance**, where the top singular subspace of the parameter update matrix captures nearly all reasoning improvements, and **Rank-1 Linear Dynamics**, where this subspace evolves linearly across training, allowing accurate prediction from early checkpoints. Based on these insights, we propose **AlphaRL**, a plug-in acceleration framework that extrapolates final parameter updates from a short early training window, achieving up to 2.5× speedup while retaining over 96% of reasoning performance.
-This model and code are provided to support further research on interpretable and efficient RL training for LLMs. Our full codebase is available at: [AlphaRL GitHub](https://github.com/caiyuchen-ustc/Alpha-RL)
 ## 🔧 Prompt Format (Chat Template)
@@ -44,8 +53,8 @@ prompt = tokenizer.apply_chat_template(
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-model = AutoModelForCausalLM.from_pretrained("caiyuchen/DAPO-step-{i}")
-tokenizer = AutoTokenizer.from_pretrained("caiyuchen/DAPO-step-{i}")
 question = "Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $(r,\theta),$ where $r > 0$ and $0 \le \theta < 2 \pi.$"
 question_with_instruction = question + "

 # On Predictability of Reinforcement Learning Dynamics for Large Language Models
+![Overview](overview.png)
 This repository provides one of the models used in our paper **"On Predictability of Reinforcement Learning Dynamics for Large Language Models"** for evaluating and predicting reinforcement learning (RL) dynamics in large language models (LLMs).
 Recent advances in LLM reasoning capabilities are largely driven by RL, yet the parameter dynamics during RL training remain poorly understood. Our work identifies two key properties of RL-induced parameter updates: **Rank-1 Dominance**, where the top singular subspace of the parameter update matrix captures nearly all reasoning improvements, and **Rank-1 Linear Dynamics**, where this subspace evolves linearly across training, allowing accurate prediction from early checkpoints. Based on these insights, we propose **AlphaRL**, a plug-in acceleration framework that extrapolates final parameter updates from a short early training window, achieving up to 2.5× speedup while retaining over 96% of reasoning performance.
+This model is one of the training checkpoints used in our paper and is provided to support research on evaluating and predicting parameter dynamics during RL training of LLMs. The full codebase is available at: [AlphaRL GitHub](https://github.com/caiyuchen-ustc/Alpha-RL).
 ## 🔧 Prompt Format (Chat Template)
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained("caiyuchen/DAPO-step-2")
+tokenizer = AutoTokenizer.from_pretrained("caiyuchen/DAPO-step-2")
 question = "Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $(r,\theta),$ where $r > 0$ and $0 \le \theta < 2 \pi.$"
 question_with_instruction = question + "