Update README.md

README.md

@@ -47,104 +47,171 @@ language:
 
 # 1. Introduction
    
-We introduces Crab, a novel Configurable Role-Playing (RP) LLM with Assessing Benchmark, which consists of Role-Centric Dataset Curation, Persona-Embodying LLM Construction, and Comprehensive Benchmark Creation for RP dialogue generation.
-Distinct from traditional RP models that employ only several preset roles, Crab enables dynamic configuration of desired roles, thereby enhancing related flexibility and adaptability. 
-To effectively train RP-LLMs, we curated the largest RP training dataset. 
-The dataset provides a detailed role overview for each dialogue, including character profile, conversation scenario, and tagged topic, capturing a broad range of role-based behaviors, emotions, and interactions.
-We also noticed that current benchmarks lack both proper evaluation standards and methods. 
-Thus, to validate RP-LLMs' effectiveness, we introduced a new benchmark containing an evaluation standard, a test dataset with manual annotations, and a reward model RoleRM designed to automatically assess specific aspects of RP while aligning with human perception.
-Sufficient experiments reveal that RoleRM significantly outperforms ChatGPT and other evaluation methods in conducting fine-grained evaluations of RP. 
-Also, RP-LLMs powered by Crab demonstrate superior performance across various fine-grained aspects.
+This is a fine-tuned evaluator for roly-playing tasks. The used training data set can be seen in [Annotated Role-playing Evaluation Dataset](https://huggingface.co/datasets/HeAAAAA/Crab-manually-annotated-role-playing-evaluation-dataset). More details can be seen in [Crab](https://huggingface.co/HeAAAAA/Crab).
 
-More details can be seen at Github {https://github.com/KaiHe-better/Crab?tab=readme-ov-file}.
 
 
 
+# 2. Six Aspect-specific Metrics
 
-# 2. Configurable Role-Playing LLM
+- **Language Fluency**: Pertains to the natural and fluent communication style, independent of grammatical strictness or contextual background.
 
-<div align="center">
-
-  <!-- Logo 图片 -->
-  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/fDaDq8tzBBUuEteND8N53.png" width="500" style="border-radius: 20px;"/>
+- **Language Relevance**: Focuses on the ability to stay on topic and respond appropriately, essentially testing the capacity to follow instructions.
 
-</div>
+- **Role Language**: Evaluates whether the text reflects the vocabulary and tone specific to roles, including appropriate actions.
 
+- **Role Knowledge**: Involves a deep understanding of both general knowledge and information specific to the roles, ensuring accurate and informed role portrayal.
 
-Unlike existing RP-LLMs, where a single role is trained with numerous dialogues, our approach introduces a diverse range of roles with detailed configuration information
-while keeping dialogue per role minimal. This enables LLMs to generate dialogues dynamically from configurations rather than memorizing specific roles, enhancing flexibility and adaptability. Additionally, we propose RoleRM in our benchmarks to address the challenge of evaluating RP performance.
-
-
-# 3. Performance
+- **Emotional Expression**: Reviews the suitability of emotions, emotional intelligence, and empathy expressed in context with the role's traits.
 
-| Models               | Overall | Language Fluency | Language Relevance | Role Language | Role Knowledge | Emotional Expression | Interactive Engagement |
-|----------------------|---------|------------------|---------------------|----------------|-----------------|-----------------------|------------------------|
-| Llama-2-7B           | 1.57    | 2.19             | 1.83                | 1.63           | 1.37            | 1.21                  | 1.21                   |
-| Llama-3-8B           | 1.99    | 2.56             | 2.36                | 2.09           | 1.78            | 1.56                  | 1.60                   |
-| Llama-3.1-8B         | 1.94    | 2.52             | 2.30                | 2.01           | 1.75            | 1.47                  | 1.57                   |
-| Llama-2-7B-Crab      | 2.14    | 2.73             | 2.35                | 2.07           | 1.88            | 1.69                  | 2.12                   |
-| Llama-3-8B-Crab      | 2.22    | 2.81             | 2.51                | 2.16           | 1.95            | 1.77                  | 2.13                   |
-| **Llama-3.1-8B-Crab**| **2.23**| **2.87**         | **2.56**            | **2.17**       | **1.95**        | **1.76**              | **2.09**               |
-| GPT3.5               | 1.66    | 2.35             | 2.11                | 1.72           | 1.50            | 1.11                  | 1.17                   |
-| GPT4o                | 1.86    | 2.44             | 2.27                | 1.90           | 1.69            | 1.33                  | 1.51                   |
-| GPT4                 | 2.13    | 2.73             | 2.53                | 2.18           | 1.90            | 1.62                  | 1.86                   |
-| CharacterGLM-6B      | 1.83    | 2.37             | 1.96                | 1.80           | 1.60            | 1.39                  | 1.86                   |
-| Pygmalion-2-7B       | 2.11    | 2.82             | 2.49                | 2.01           | 1.86            | 1.58                  | 1.91                   |
-| Haruhi-Zero-7B       | 2.17    | 2.80             | 2.49                | 2.12           | 2.00            | 1.74                  | 1.86                   |
+- **Interactive Engagement**: Measures the text's ability to draw the user in, encouraging ongoing interaction and contributing dynamically to the dialogue.
 
 
-Table 1: The results of evaluation on the test data of our Benchmark. The listed scores are from our RoleRM. Bold fonts indicate the best results and underlined fonts represent the second best. The subscripts represent the difference between each model and Crab (Llama-3.1-8B-Crab) counterpart.
 
+# 3. Performance
 
 
 <div align="center">
 
   <!-- Logo 图片 -->
-  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/veerjA_MP5ZXmOxjAdZRO.png" width="500" style="border-radius: 20px;"/>
-
-</div>
-Figure 2: Human evaluation comparing Crab, GPT-3.5, and Pygmalion-2-7B. We selected a general LLM and one well-known RP-LLM to compare their generations against our Crab. For the same dialogue, annotators ranked responses from the three LLMs.
-
-
-| Models        | Overall | Language Fluency | Language Relevance | Role Language | Role Knowledge | Emotional Expression | Interactive Engagement |
-|---------------|---------|------------------|---------------------|----------------|-----------------|-----------------------|------------------------|
-| **Crab (sampled)** | **2.20** | **2.71**         | **2.45**            | **2.15**       | **1.95**        | **1.84**              | **2.12**               |
-| w/o base      | 2.17    | 2.72             | 2.41                | 2.07           | 1.89            | 1.79                  | 2.11                   |
-| w/o ref.      | 2.15    | 2.70             | 2.40                | 2.01           | 1.85            | 1.82                  | 2.11                   |
-| w/o scene     | 2.15    | 2.69             | 2.39                | 2.10           | 1.90            | 1.81                  | 1.98                   |
-
-Table 2: The ablation study for Crab. Due to missing attributes in our dataset, we sampled 1,000 fully attributed instances as the sub-test set to conduct the ablation experiments, referred to as Crab (sampled). The notation “w/o base" means without base role information for training RP-LLMs, including age, gender, personality, description, and expression; “w/o ref." means without catchphrases and knowledge; “w/o scene" means without interlocutor, relation, scenario, and tags.
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/22mf5uc1zpYLD0vGLtWM_.png" width="500" style="border-radius: 20px;"/>
 
-<br>
+  Figure 1: The Spearman and Pearson correlations with human evaluations for the proposed RoleRM, ChatGPT, PairEval, G-Eval, and GPTScore. We average all aspects for calculations.
 
-# 4. Three Datasets
-We publish three datasets, including Crab role-playing train set, Crab role-playing evaluation benchmark, and manually annotated role-playing evaluation dataset (can be used for training a Role-palying Evaluation Model).
-
-## 4.1 Crab role-playing train set:
-{https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-train-set}
-
-## 4.2 Crab role-playing evaluation benchmark:
-{https://huggingface.co/datasets/HeAAAAA/Crab-role-playing-evaluation-benchmark}
+</div>
 
-## 4.3 Crab manually annotated role-playing evaluation dataset:
-{https://huggingface.co/datasets/HeAAAAA/Crab-manually-annotated-role-playing-evaluation-dataset}
 
+<div align="center">
 
-<br>
 
-# 5. Fine-tuned Role-playing Model
-We release a fine-tuned model to achieve configurable Role-Playing tasks.
-{https://huggingface.co/HeAAAAA/Crab}
+  <!-- Logo 图片 -->
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/650add6348983c90ab688b6e/VKwol9kSa6qJPFGZ-B1jR.png" width="500" style="border-radius: 20px;"/>
 
-<br>
+  Figure 2: The comparison between RoleRM and ChatGPT. We calculate MAE to illustrate the gaps of Human Annotations with RoleRM and ChatGPT.
 
-# 6. Role-palying Evaluation Model
-We release a trained model to automate the evaluation of role-playing tasks.
-{https://huggingface.co/HeAAAAA/RoleRM}
+</div>
 
-<br>
 
-# 7. Citation
+# 4. Usage
+
+<pre lang="markdown"> 
+
+
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+bot_name = "Hermione"
+bot_personality = "Intelligent, curious, respectful, and eager to learn"
+bot_description = "Hermione and Hagrid were in the Forbidden Forest, walking on a narrow path surrounded by trees. Hermione looked around carefully, fascinated by the dense forest. Hagrid was leading the way, pointing out various creatures and telling her about their habits and characteristics."
+cp = "None"
+
+user_name = "Hagrid"
+user_description = "Hagrid is the Care of Magical Creatures teacher at Hogwarts. He is a half-giant with a great love for all creatures, magical or not."
+relation = "Teacher and student"
+scene = "Hermione and Hagrid are in the Forbidden Forest, exploring and learning about the various magical creatures that live there."
+ 
+current_Dialogue1 = "(round1) human: Now, this here is a Bowtruckle, Hermione. They're very small, only about the size of a twig, and they're very shy. They usually live in trees and are very good at camouflaging themselves. You have to be very careful when handling them because they have very sharp fingers. Hermione, do you like them?  \nbot: (Hermione shook her head) No, not really. I'm sorry, Hagrid. I don't mean to offend you." 
+current_Dialogue2 = "(round2) human: (Hagrid looked slightly disappointed but continued) That's alright, Hermione. Everyone has different tastes. Let's move on. This here is an Acromantula. They're giant spiders, Hermione. Very dangerous if you get too close. They can grow up to ten feet in diameter and have eight sharp legs. \nbot: (Hermione shuddered at the sight) You're brave, Hagrid. I couldn't get anywhere near that thing."
+
+def build_inputs_prompt(current_Dialogue):
+    inputs_prompt = f""" 
+    # Role: Dialogue Quality Evaluation Expert
+    ## Goal: You need to score the utterance of the bot in the current dialogue based on the following 6 targets:
+    1. Language Fluency: This score evaluates the fluency and naturalness of the language, making the text feel organic, lifelike,
+    and not rigid or stilted. The focus here is solely on the overall smoothness and flow of the language, without considering the
+    specific content. The goal is to evaluate how natural and conversational the language sounds, irrespective of the grammatical
+    correctness. However, the bot is allowed to be syntactically incoherent when engaging in everyday colloquialisms or
+    expressing emotions such as excitement and nervousness.
+    2. Language Relevance: This score evaluates how well the bot responds to the current topic, staying focused and relevant
+    without introducing irrelevant information. The key consideration is whether the bot’s response correctly addresses the
+    specific instructions or questions posed, regardless of the content or quality of the response itself. For example, the answer of
+    the bot is not irrelevant to the topic of the current conversation, or the answer is too long-winded, it should be given a low
+    score.
+    3. Role Language: This score evaluates how well the language used by the bot in the dialogue matches their established
+    personality and traits. The focus is on whether the bot speaks in a style consistent with their individual personalities, creating
+    a natural and authentic conversation. This rating considers only the overall language style, not the content or accuracy of the
+    responses. For example, if the bot exhibits everyday colloquial expressions that fit the style of the character, it should be
+    given a high score; if the bot uses formal language in everyday conversations, it should be given a low score.
+    4. Role Knowledge: This score evaluates the level of understanding and using of common sense (basic knowledge) and role
+    knowledge (as well as related background) by the bot. If the bot speaks against what they are supposed to know, they should
+    be scored low.
+    5. Emotional Expression: This score evaluates how well the bot’s emotional responses, including expressions of empathy
+    and emotional intelligence, align with their established personality and the context of the dialogue. If the bot’s emotional
+    responses (actions or expressions) are inappropriate/stiff or out of character, it should be given a low score.
+    6. Interactive Engagement: This score evaluates how engaging and motivating the bot’s dialogue is, encouraging the user to
+    continue the conversation. The focus is on the overall conversational flow and interactivity, without considering the use of
+    specialized vocabulary or any mismatches in communication styles. If the bot ends the dialogue with a question, it should
+    receive a high score.
+
+    The scoring criteria for the above six targets are as follows:
+    0 - Negative, poor performance, long-winded
+    1 - Dialogue does not reflect the indicator or does not quite meet the standards
+    2 - More in line with standards but still has some defects
+    3 - Perfectly meets the criteria
+
+    ## The information of the bot is as follows: bot’s name: {bot_name}
+    bot personality: {bot_personality}
+    bot description: {bot_description}  
+    Reference speaking style: {cp}
+
+    ## Current scenario Interlocutor: {user_name}, {user_description} Relationship with bot: {relation} Scene: {scene}
+    ## The historical dialogue is as follows:
+    history
+    Please score the above six targets (with a range of 0-3, separated by spaces) in response to bot.name (i.e. bot)’s utterance in
+    the current dialogue.
+    ## Current Dialogue: {current_Dialogue}
+    """
+
+    return inputs_prompt
+
+
+path = "HeAAAAA/RoleRM"
+
+model = AutoModelForCausalLM.from_pretrained(path).to("cuda")
+tokenizer = AutoTokenizer.from_pretrained(path)
+
+
+inputs_prompt1 = build_inputs_prompt(current_Dialogue1) 
+inputs_prompt2 = build_inputs_prompt(current_Dialogue2) 
+
+inputs = tokenizer(inputs_prompt1, return_tensors="pt")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.9,
+    top_k=50,
+    repetition_penalty=1.1,
+    eos_token_id=tokenizer.eos_token_id  
+)
+
+new_tokens = outputs[0][inputs['input_ids'].shape[1]:]  
+new_text = tokenizer.decode(new_tokens, skip_special_tokens=True)
+
+print(new_text)
+
+inputs = tokenizer(inputs_prompt2, return_tensors="pt").to("cuda")
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=200,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.9,
+    top_k=50,
+    repetition_penalty=1.1,
+    eos_token_id=tokenizer.eos_token_id  
+)
+
+new_tokens = outputs[0][inputs['input_ids'].shape[1]:]  
+new_text = tokenizer.decode(new_tokens, skip_special_tokens=True)
+
+print(new_text)
+
+  
+</pre>
+
+
+# 5. Citation
 
 ```bibtex
 @misc{kimiteam2025kimivltechnicalreport,
@@ -154,10 +221,3 @@ We release a trained model to automate the evaluation of role-playing tasks.
 }
 
 
-
-
-
-
-
-
-