Update README.md
Browse files
README.md
CHANGED
|
@@ -16,5 +16,5 @@ base_model:
|
|
| 16 |
|
| 17 |
Recent advances in large language models (LLMs) have highlighted the potential of reinforcement learning with verifiable rewards (RLVR) to enhance reasoning capabilities through extended output sequences. However, traditional RL frameworks face inefficiencies when handling ultra-long outputs due to long-tail sequence distributions and entropy collapse during training. To address these challenges, we propose an <strong>U</strong>ltra-<strong>L</strong>ong <strong>O</strong>utput <strong>R</strong>einforcement <strong>L</strong>earning (UloRL) approach for advancing large language models' reasoning abilities. Specifically, we divide ultra long output decoding into short segments, enabling efficient training by mitigating delays caused by long-tail samples. Additionally, we introduce dynamic masking of well-<strong>M</strong>astered <strong>P</strong>ositive <strong>T</strong>okens (MPTs) to prevent entropy collapse. Experimental results demonstrate the effectiveness of our approach. On the Qwen3-30B-A3B model, RL with segment rollout achieved 2.06x increase in training speed, while RL training with 128k-token outputs improves the model's performance on AIME2025 from 70.9\% to 85.1\% and on BeyondAIME from 50.7\% to 61.9\%, even surpassing Qwen3-235B-A22B with remarkable gains.
|
| 18 |
<div align="center">
|
| 19 |
-
<img src="
|
| 20 |
</div>
|
|
|
|
| 16 |
|
| 17 |
Recent advances in large language models (LLMs) have highlighted the potential of reinforcement learning with verifiable rewards (RLVR) to enhance reasoning capabilities through extended output sequences. However, traditional RL frameworks face inefficiencies when handling ultra-long outputs due to long-tail sequence distributions and entropy collapse during training. To address these challenges, we propose an <strong>U</strong>ltra-<strong>L</strong>ong <strong>O</strong>utput <strong>R</strong>einforcement <strong>L</strong>earning (UloRL) approach for advancing large language models' reasoning abilities. Specifically, we divide ultra long output decoding into short segments, enabling efficient training by mitigating delays caused by long-tail samples. Additionally, we introduce dynamic masking of well-<strong>M</strong>astered <strong>P</strong>ositive <strong>T</strong>okens (MPTs) to prevent entropy collapse. Experimental results demonstrate the effectiveness of our approach. On the Qwen3-30B-A3B model, RL with segment rollout achieved 2.06x increase in training speed, while RL training with 128k-token outputs improves the model's performance on AIME2025 from 70.9\% to 85.1\% and on BeyondAIME from 50.7\% to 61.9\%, even surpassing Qwen3-235B-A22B with remarkable gains.
|
| 18 |
<div align="center">
|
| 19 |
+
<img src="home.jpg" width="80%" />
|
| 20 |
</div>
|