Enhance model card with metadata, paper, code, and usage

README.md

@@ -1,8 +1,134 @@
 ---
+base_model:
+- Qwen/Qwen2.5-7B-Instruct
 license: other
 license_name: license
 license_link: LICENSE
-base_model:
-- Qwen/Qwen2.5-7B-Instruct
+pipeline_tag: text-generation
+library_name: transformers
 ---
-https://www.arxiv.org/abs/2507.03112
+
+# RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
+
+This repository contains the official model for the paper [RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents](https://huggingface.co/papers/2507.03112).
+
+<div align="center">
+  [![Model](https://img.shields.io/badge/Model-4d5eff?style=for-the-badge&logo=huggingface&logoColor=ffffff&labelColor)](https://huggingface.co/RLVER)
+  [![Github](https://img.shields.io/badge/Code-000000?style=for-the-badge&logo=github&logoColor=white)](https://github.com/Tencent/digitalhuman/tree/main/RLVER)
+  [![arXiv](https://img.shields.io/badge/arXiv-2507.03112-b31b1b.svg?style=for-the-badge)](https://arxiv.org/abs/2507.03112)
+</div>
+
+## Abstract
+Large language models (LLMs) excel at logical and algorithmic reasoning, yet their emotional intelligence (EQ) still lags far behind their cognitive prowess. While reinforcement learning from verifiable rewards (RLVR) has advanced in other domains, its application to dialogue-especially for emotional intelligence-remains underexplored. In this work, we introduce RLVER, the first end-to-end reinforcement learning framework that leverages verifiable emotion rewards from simulated users to cultivate higher-order empathetic abilities in LLMs. Within this framework, self-consistent affective simulated users engage in dialogue rollouts and produce deterministic emotion scores during conversations, serving as reward signals to guide the LLM's learning. Fine-tuning publicly available Qwen2.5-7B-Instruct model with PPO boosts its Sentient-Benchmark score from 13.3 to 79.2 while largely preserving mathematical and coding competence. Extensive experiments reveal that: (i) RLVER consistently improves multiple dialogue capabilities; (ii) Thinking and non-thinking models show distinct trends--thinking models excel in empathy and insight, while non-thinking models favor action; (iii) GRPO often yields stable gains, while PPO can push certain capabilities to a higher ceiling; (iv) More challenging environments are not always better-moderate ones can yield stronger outcomes. Our results show that RLVER is a practical route toward emotionally intelligent and broadly capable language agents.
+
+## Overview
+<p align="center"><img width="90%" src="https://github.com/Tencent/digitalhuman/raw/main/RLVER/code/figs/framework.png" alt="RLVER Framework" /></p>
+<p align="center"><em>The overview of RLVER. In this work, we present the first end-to-end reinforcement-learning framework that equips an LLM with human-level empathetic skills by optimizing against verifiable emotion rewards.</em></p>
+
+<p align="center"><img width="90%" src="https://github.com/Tencent/digitalhuman/raw/main/RLVER/code/figs/main_result.png" alt="RLVER Main Result" /></p>
+<p align="center"><em>The main result of RLVER.</em></p>
+
+## Usage
+
+You can load the model using the `transformers` library. Below is a quick example of how to use it for text generation:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers.generation import GenerationConfig
+
+# Load model and tokenizer
+# Replace "{your_model_id}" with the actual model ID (e.g., "Tencent/RLVER")
+model = AutoModelForCausalLM.from_pretrained(
+    "{your_model_id}", 
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True
+)
+tokenizer = AutoTokenizer.from_pretrained("{your_model_id}", trust_remote_code=True)
+model.eval()
+
+# Configure generation
+generation_config = GenerationConfig.from_pretrained(
+    "{your_model_id}",
+    trust_remote_code=True
+)
+generation_config.chat_format = "chatml"
+
+# Example chat interaction
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "Hello! How are you today?"}
+]
+
+text = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True
+)
+
+# Generate response
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+generated_ids = model.generate(
+    model_inputs.input_ids,
+    attention_mask=model_inputs.attention_mask,
+    generation_config=generation_config
+)
+
+# Decode and print only the generated part
+generated_ids = [
+    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(generated_text)
+```
+
+## Environment Setup 🔧
+
+```bash
+pip install -r requirements.txt
+```
+This project uses Ray and Verl frameworks. You should configure your settings in `train_rlver.sh` before the training.
+
+## Data 📚
+The training and test user profiles are provided in `data/test_profile.jsonl`.
+
+## Start Training 🔥
+1.  **Basic Settings**
+    Set the data path, model path, and other configurations.
+    To enable "Think then say" mode during training, set `IF_THINK=True`.
+    To use GRPO, set `algorithm.adv_estimator=grpo` and `actor_rollout_ref.rollout.n=4`.
+2.  **Set up RAY**
+    First set up the RAY environment.
+    ```bash
+    # On chief node:
+    ray start --head --port=$PORT_ID --node-ip-address=$CHIEF_IP
+
+    # On each worker node:
+    ray start --address=$CHIEF_IP:$PORT_ID --node-ip-address=$CURRENT_IP
+    ```
+3.  **Start training**
+    Now you can run `train_rlver.sh` to start training!
+    ```bash
+    sh ./train_rlver.sh
+    ```
+
+## Evaluation 🏆
+To evaluate your trained model:
+1.  **Convert to Hugging Face format**:
+    ```bash
+    sh ./mk_hf_model.sh
+    ```
+2.  **Evaluate with SAGE**
+    You can evaluate the converted model following the instructions in [SAGE](https://github.com/Tencent/digitalhuman/tree/main/SAGE).
+
+## Citation
+If you find our work helpful or inspiring, please feel free to cite it.
+```bibtex
+@article{zhou2025rlver,
+  title={RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents},
+  author={Zhou, Zijian and Liu, Shikun and Han, Xiao and Liu, Haozhe and Ng, Kam Woh and Xie, Tian and Cong, Yuren and Li, Hang and Xu, Mengmeng and Pérez-Rúa, Juan-Manuel and Patel, Aditya and Xiang, Tao and Shi, Miaojing and He, Sen},
+  journal={arXiv preprint arXiv:2507.03112},
+  year={2025},
+}
+```