Update README.md

README.md

@@ -1,6 +1,136 @@
 ---
 license: apache-2.0
+language:
+- en
+base_model:
+- meta-llama/Llama-3.1-8B
 ---
 
 
-The fine-tuned Role-playing LLM from the paper "Crab: A Novel Configurable Role-Playing LLM with Assessing Benchmark"
+
+
+<div align="center">
+
+  <!-- Logo 图片 -->
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/zT-y1dEjenfRwGk0ezjGc.png" width="300" style="border-radius: 20px;"/>
+
+  <!-- 链接部分 -->
+  <p style="margin-top: 20px;">
+    <a href="https://example.com/report" style="margin: 0 10px;">
+      📄 <strong>Paper</strong>
+    </a> |
+    <a href="https://github.com/KaiHe-better/Crab?tab=readme-ov-file" style="margin: 0 10px;">
+      📄 <strong>Github</strong>
+    </a> |
+     <a href="https://huggingface.co/HeAAAAA/RoleRM" style="margin: 0 10px;">
+      💬 <strong>Role-palying Evaluation Model</strong>
+    </a>
+    <br>
+    <a href="https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-train-set" style="margin: 0 10px;">
+      💬 <strong>Training Dataset</strong>
+    </a>  |
+    <a href="https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-evaluation-benchmark" style="margin: 0 10px;">
+      💬 <strong>Evaluation Benchmark</strong>
+    </a>  |
+    <a href="https://huggingface.co/datasets/HeAAAAA/Crab-manually-annotated-role-playing-evaluation-dataset" style="margin: 0 10px;">
+      💬 <strong>Annotated Role-playing Evaluation Dataset</strong>
+    </a>
+    
+  </p>
+
+</div>
+
+
+
+
+# 1. Introduction
+   
+We introduces Crab, a novel Configurable Role-Playing (RP) LLM with Assessing Benchmark, which consists of Role-Centric Dataset Curation, Persona-Embodying LLM Construction, and Comprehensive Benchmark Creation for RP dialogue generation.
+Distinct from traditional RP models that employ only several preset roles, Crab enables dynamic configuration of desired roles, thereby enhancing related flexibility and adaptability. 
+To effectively train RP-LLMs, we curated the largest RP training dataset. 
+The dataset provides a detailed role overview for each dialogue, including character profile, conversation scenario, and tagged topic, capturing a broad range of role-based behaviors, emotions, and interactions.
+We also noticed that current benchmarks lack both proper evaluation standards and methods. 
+Thus, to validate RP-LLMs' effectiveness, we introduced a new benchmark containing an evaluation standard, a test dataset with manual annotations, and a reward model RoleRM designed to automatically assess specific aspects of RP while aligning with human perception.
+Sufficient experiments reveal that RoleRM significantly outperforms ChatGPT and other evaluation methods in conducting fine-grained evaluations of RP. 
+Also, RP-LLMs powered by Crab demonstrate superior performance across various fine-grained aspects.
+
+More details can be seen at Github {https://github.com/KaiHe-better/Crab?tab=readme-ov-file}.
+
+# 2. Configurable Role-Playing LLM
+
+<div align="center">
+
+  <!-- Logo 图片 -->
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/fDaDq8tzBBUuEteND8N53.png" width="500" style="border-radius: 20px;"/>
+
+</div>
+
+
+Unlike existing RP-LLMs, where a single role is trained with numerous dialogues, our approach introduces a diverse range of roles with detailed configuration information
+while keeping dialogue per role minimal. This enables LLMs to generate dialogues dynamically from configurations rather than memorizing specific roles, enhancing flexibility and adaptability. Additionally, we propose RoleRM in our benchmarks to address the challenge of evaluating RP performance.
+
+
+# 3. Performance
+
+| Models               | Overall | Language Fluency | Language Relevance | Role Language | Role Knowledge | Emotional Expression | Interactive Engagement |
+|----------------------|---------|------------------|---------------------|----------------|-----------------|-----------------------|------------------------|
+| Llama-2-7B           | 1.57    | 2.19             | 1.83                | 1.63           | 1.37            | 1.21                  | 1.21                   |
+| Llama-3-8B           | 1.99    | 2.56             | 2.36                | 2.09           | 1.78            | 1.56                  | 1.60                   |
+| Llama-3.1-8B         | 1.94    | 2.52             | 2.30                | 2.01           | 1.75            | 1.47                  | 1.57                   |
+| Llama-2-7B-Crab      | 2.14    | 2.73             | 2.35                | 2.07           | 1.88            | 1.69                  | 2.12                   |
+| Llama-3-8B-Crab      | 2.22    | 2.81             | 2.51                | 2.16           | 1.95            | 1.77                  | 2.13                   |
+| **Llama-3.1-8B-Crab**| **2.23**| **2.87**         | **2.56**            | **2.17**       | **1.95**        | **1.76**              | **2.09**               |
+| GPT3.5               | 1.66    | 2.35             | 2.11                | 1.72           | 1.50            | 1.11                  | 1.17                   |
+| GPT4o                | 1.86    | 2.44             | 2.27                | 1.90           | 1.69            | 1.33                  | 1.51                   |
+| GPT4                 | 2.13    | 2.73             | 2.53                | 2.18           | 1.90            | 1.62                  | 1.86                   |
+| CharacterGLM-6B      | 1.83    | 2.37             | 1.96                | 1.80           | 1.60            | 1.39                  | 1.86                   |
+| Pygmalion-2-7B       | 2.11    | 2.82             | 2.49                | 2.01           | 1.86            | 1.58                  | 1.91                   |
+| Haruhi-Zero-7B       | 2.17    | 2.80             | 2.49                | 2.12           | 2.00            | 1.74                  | 1.86                   |
+
+
+Table 1: The results of evaluation on the test data of our Benchmark. The listed scores are from our RoleRM. Bold fonts indicate the best results and underlined fonts represent the second best. The subscripts represent the difference between each model and Crab (Llama-3.1-8B-Crab) counterpart.
+
+
+
+<div align="center">
+
+  <!-- Logo 图片 -->
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/veerjA_MP5ZXmOxjAdZRO.png" width="500" style="border-radius: 20px;"/>
+
+</div>
+Figure 2: Human evaluation comparing Crab, GPT-3.5, and Pygmalion-2-7B. We selected a general LLM and one well-known RP-LLM to compare their generations against our Crab. For the same dialogue, annotators ranked responses from the three LLMs.
+
+
+| Models        | Overall | Language Fluency | Language Relevance | Role Language | Role Knowledge | Emotional Expression | Interactive Engagement |
+|---------------|---------|------------------|---------------------|----------------|-----------------|-----------------------|------------------------|
+| **Crab (sampled)** | **2.20** | **2.71**         | **2.45**            | **2.15**       | **1.95**        | **1.84**              | **2.12**               |
+| w/o base      | 2.17    | 2.72             | 2.41                | 2.07           | 1.89            | 1.79                  | 2.11                   |
+| w/o ref.      | 2.15    | 2.70             | 2.40                | 2.01           | 1.85            | 1.82                  | 2.11                   |
+| w/o scene     | 2.15    | 2.69             | 2.39                | 2.10           | 1.90            | 1.81                  | 1.98                   |
+
+Table 2: The ablation study for Crab. Due to missing attributes in our dataset, we sampled 1,000 fully attributed instances as the sub-test set to conduct the ablation experiments, referred to as Crab (sampled). The notation “w/o base" means without base role information for training RP-LLMs, including age, gender, personality, description, and expression; “w/o ref." means without catchphrases and knowledge; “w/o scene" means without interlocutor, relation, scenario, and tags.
+
+# 4. Three Datasets
+We publish three datasets, including Crab role-playing train set, Crab role-playing evaluation benchmark, and manually annotated role-playing evaluation dataset (can be used for training a Role-palying Evaluation Model).
+
+Crab role-playing train set:
+{https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-train-set}
+
+Crab role-playing evaluation benchmark:
+{https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-evaluation-benchmark}
+
+Crab manually annotated role-playing evaluation dataset:
+{https://huggingface.co/datasets/HeAAAAA/Crab-manually-annotated-role-playing-evaluation-dataset}
+
+# 5. Role-palying Evaluation Model
+We also release a trained model to automate the evaluation of role-playing tasks.
+{https://huggingface.co/HeAAAAA/RoleRM}
+
+# 6. Citation
+
+```bibtex
+@misc{kimiteam2025kimivltechnicalreport,
+  title={Crab: A Novel Configurable Role-Playing LLM with Assessing Benchmark},
+  author={Kai He, Yucheng Huang, Wenqing Wang, et.al.},
+  year={2025},
+}