Improve model card: Add tags, abstract, and paper link

README.md

@@ -1,8 +1,14 @@
 ---
 license: apache-2.0
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- code-generation
 ---
 
+# ASPO: Asymmetric Importance Sampling Policy Optimization
 
+This model checkpoint is for the paper [ASPO: Asymmetric Importance Sampling Policy Optimization](https://huggingface.co/papers/2510.06062).
 
 <div align="center">
 
@@ -27,6 +33,10 @@ license: apache-2.0
 
 </div>
 
+## Abstract
+
+Recent Large Language Model (LLM) post-training methods rely on token-level clipping mechanisms during Reinforcement Learning (RL). However, we identify a fundamental flaw in this Outcome-Supervised RL (OSRL) paradigm: the Importance Sampling (IS) ratios of positive-advantage tokens are mismatched, leading to unbalanced token weighting for positive and negative tokens. This mismatch suppresses the update of low-probability tokens while over-amplifying already high-probability ones. To address this, we propose Asymmetric Importance Sampling Policy Optimization (ASPO), which uses a simple yet effective strategy that flips the IS ratios of positive-advantage tokens, aligning their update direction with the learning dynamics of negative ones. AIS further incorporates a soft dual-clipping mechanism to stabilize extreme updates while maintaining gradient flow. Comprehensive experiments on coding and mathematical reasoning benchmarks demonstrate that ASPO significantly mitigates premature convergence, improves training stability, and enhances final performance over strong GRPO-based baselines. Our analysis provides new insights into the role of token-level weighting in OSRL and highlights the critical importance of correcting IS in LLM RL. The code and models of ASPO are available at this https URL .
+
 ## Overview
 
 **Archer2.0** marks a significant evolution from its predecessor through the introduction of **Asymmetric Importance Sampling Policy Optimization (ASPO)**, which is designed to overcome the fundamental limitations of **PPO-Clip**, effectively mitigating issues like **entropy collapse** and **repetitive outputs**, preventing **premature convergence**, and thereby enabling more advanced **reinforcement learning** capabilities.
@@ -103,3 +113,120 @@ While our mathematical models are still in training and have not converged, we h
     </tr>
   </tbody>
 </table>
+
+
+## Getting Started
+
+### 1 Installation
+
+```bash
+# Installing Python 3.10 Environment.
+conda create -n archer python=3.10 -y
+conda activate archer
+
+# Installing dependencies.
+pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu124
+wget -nv https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
+pip install --no-cache-dir flash_attn-2.7.3+cu12torch2.5cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
+
+cd Archer2.0
+pip install -e .
+```
+
+#### Initialize Ray Cluster
+
+We have provided a one-click script to initialize Ray environments on any number of machines. Run the following command on the head node:
+
+```bash
+bash ./tools/start_ray.sh
+```
+
+Note:
+- Please replace your_wandb_api_key in export WANDB_API_KEY=your_wandb_api_key with your actual key.
+- Hostfile locations vary across operating systems (e.g., on my machine, it's located at /etc/mpi/hostfile). Locate the file on your server and modify its content accordingly.
+
+### 2 Training
+
+We have currently provided the script and data to reproduce the results of the “Archer2.0-Code-1.5B-Preview”.
+
+```bash
+bash ./scripts/train/run_archer2.0_qwen2.5_1.5b_code.sh
+```
+
+### 3 Evaluation
+
+When using the Verl framework for RL training, we observed a consistent discrepancy between the evaluation results produced by the in-training weights and the saved model checkpoints. To ensure the accurate selection of model checkpoints, our evaluation is conducted using the saved checkpoints.
+
+#### 3.1 Automated Evaluation Pipeline
+To automatically scan a specified directory and evaluate all saved model checkpoints during training, run the following script on a GPU-enabled machine:
+
+```bash
+bash ./tools/run_eval_pipeline.sh
+```
+Since code evaluation tasks run on CPU only, we separate the LiveCodeBench evaluation to optimize GPU utilization. Execute the following script on a CPU machine to automatically evaluate the inference results generated in the previous step:
+
+```bash
+bash ./tools/run_lcb_eval.sh
+```
+
+#### 3.2 Head-On Evaluation
+
+##### Step 1: Convert Model Format
+
+Run the following command to convert the model to Hugging Face format:
+
+```bash
+bash ./tools/model_merge.sh
+```
+
+##### Step 2: Run Inference
+
+Execute the script below to generate inference results for the test data:
+
+```bash
+bash ./scripts/eval/run_eval.sh
+```
+
+##### Step 3: Run Evaluation
+
+Navigate to line 245 in [LiveCodeBench/blob/main/lcb_runner/evaluation/compute_code_generation_metrics_v5.py](https://github.com/wizard-III/LiveCodeBench/blob/main/lcb_runner/evaluation/compute_code_generation_metrics_v5.py#L245) and update the parquet_file path to point to the result file generated in Step 2.
+
+Execute the following script to evaluate performance on the LiveCodeBench v5 benchmark:
+
+```bash
+python LiveCodeBench/lcb_runner/evaluation/compute_code_generation_metrics_v5.py
+```
+
+Note: Please update the path parameters in the scripts above as needed.
+
+## Technical Report
+[ASPO: Asymmetric Importance Sampling Policy Optimization](https://huggingface.co/papers/2510.06062)
+
+## Acknowledgements
+
+- We build our model upon [`DeepSeek-R1-Distill-Qwen-1.5B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B).
+- Training was carried out with a modified version of [verl](https://github.com/volcengine/verl).
+
+## Citation
+
+Please cite the following:
+```bibtex
+@misc{wang2025aspoasymmetricimportancesampling,
+      title={ASPO: Asymmetric Importance Sampling Policy Optimization},
+      author={Jiakang Wang and Runze Liu and Lei Lin and Wenping Hu and Xiu Li and Fuzheng Zhang and Guorui Zhou and Kun Gai},
+      year={2025},
+      eprint={2510.06062},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2510.06062},
+}
+```
+
+```bibtex
+@article{wang2025stabilizing,
+  title={Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR},
+  author={Wang, Jiakang and Liu, Runze and Zhang, Fuzheng and Li, Xiu and Zhou, Guorui},
+  journal={arXiv preprint arXiv:2507.15778},
+  year={2025}
+}
+```