Add Model Card

README.md

@@ -11,6 +11,47 @@ tags:
 arxiv: 2412.17743
 model-index:
 - name: YuLan-Mini-Instruct
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      type: openai_humaneval
+      name: HumanEval
+    metrics:
+    - name: pass@10
+      type: pass@10
+      value: 0.866
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: mbpp
+      name: MBPP
+    metrics:
+    - name: pass@10
+      type: pass@10
+      value: 0.857
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: math
+      name: MATH
+    metrics:
+    - name: maj@1
+      type: maj@1
+      value: 0.552
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: gsm8k
+      name: GSM8K
+    metrics:
+    - name: maj@1
+      type: maj@1
+      value: 0.717
+      verified: false
 ---
 
 # Important Notice: This is the **post-training version** of YuLan-Mini.
@@ -183,11 +224,16 @@ We utilize the metrics outlined in following table, where higher scores are cons
 |                | MMLU<sub> (5 shot)</sub>       |
 |                | TruthfulQA<sub>(0 shot)</sub>  |
 | **Reasoning**  | ARC<sub>(0 shot)</sub>         |
+|                | GPQA<sub>(5 shot)</sub>        |
 | **Math**       | MATH<sub>(0 shot, CoT)</sub>   |
 |                | GSM8K<sub>(8 shot)</sub>       |
 |                | GSM8K<sub>(0 shot, CoT)</sub>  |
-| **Code**       | HumanEval<sub>(pass@10)</sub>  |
+| **Code**       | HumanEval<sub>(pass@1)</sub>   |
+|                | HumanEval+<sub>(pass@1)</sub>  |
+|                | HumanEval<sub>(pass@10)</sub>  |
 |                | HumanEval+<sub>(pass@10)</sub> |
+|                | MBPP<sub>(pass@1)</sub>        |
+|                | MBPP+<sub>(pass@1)</sub>       |
 |                | MBPP<sub>(pass@10)</sub>       |
 |                | MBPP+<sub>(pass@10)</sub>      |
 
@@ -216,13 +262,18 @@ We evaluate code generation capabilities across four established benchmarks: Hum
 | MMLU<sub> (5 shot)</sub>       | 52.7                   | 63.4                     | **66.5**                   |
 | TruthfulQA<sub>(0 shot)</sub>  | 50.1                   | 49.7                     | **58.8**                   |
 | ARC<sub>(0 shot)</sub>         | 51.8                   | **78.6**                 | 47.8                       |
-| MATH<sub>(0 shot, CoT)</sub>   | 55.2                   | 48                       | **55.2**                   |
+| GPQA<sub>(5 shot)</sub>        | 30.1                   | **32.6**                 | 29.8                   |
+| MATH<sub>(0 shot, CoT)</sub>   | **55.2**               | 48.0                       | **55.2**                   |
 | GSM8K<sub>(8 shot)</sub>       | **81.8**                | 43.4                     | 73.2                       |
-| GSM8K<sub>(0 shot, CoT)</sub>  | **71.8**               | 66                       | 69.4                       |
+| GSM8K<sub>(0 shot, CoT)</sub>  | **71.7**               | 66.0                       | 69.4                       |
+| HumanEval<sub>(pass@1)</sub>   | **67.7**                | 51.5                     | 61.6                       |
+| HumanEval+<sub>(pass@1)</sub>  | **61.6**                | 45.2                     | 47.0                       |
 | HumanEval<sub>(pass@10)</sub>  | **86.6**                | 78.7                     | 84.1                       |
-| HumanEval+<sub>(pass@10)</sub> | **80.5**                | 72                       | 78.0                       |
+| HumanEval+<sub>(pass@10)</sub> | **80.5**                | 72.0                       | 78.0                       |
+| MBPP<sub>(pass@1)</sub>        | **66.7**                | 57.4                     | 63.2                       |
+| MBPP+<sub>(pass@1)</sub>       | **56.6**                | 47.8                     | 52.0                       |
 | MBPP<sub>(pass@10)</sub>       | 85.7                    | 80.4                     | **88.1**                   |
-| MBPP+<sub>(pass@10)</sub>      | 75.4                    | 71.2                     | 77.5                       |
+| MBPP+<sub>(pass@10)</sub>      | 75.4                    | 71.2                     | **77.5**                       |
 
 
 ## 7. Conclusion, Limitation, and Future Work
@@ -231,6 +282,46 @@ We evaluate code generation capabilities across four established benchmarks: Hum
 
 We propose YuLan-Mini-Instruct, a powerful small-scale language model with 2.4 billion parameters with complete post-training process with SFT, DPO and PPO strategies. Although YuLan-Mini-Instruct demonstrates limitations on knowledge-intensive benchmarks such as MMLU, our experimental results indicate that it exhibit competitive performance in several general-purpose tasks. We anticipate that our empirical contributions will contribute to the development of more robust and generalizable LLMs. Future research directions will focus on enhancing dataset diversity and comprehensiveness through expanded training data collection to improve reasoning capabilities. 
 
+## Quick Start 💻
+
+Below is a simple example for inference using Huggingface:
+
+**Huggingface Inference Example**
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("yulan-team/YuLan-Mini-Instruct")
+model = AutoModelForCausalLM.from_pretrained("yulan-team/YuLan-Mini-Instruct", torch_dtype=torch.bfloat16)
+
+# Input text
+chat = [
+    {"role": "system", "content": "You are YuLan-Mini, created by RUC AI Box. You are a helpful assistant."},
+    {"role": "user", "content": "What is Renmin University of China?"}
+]
+formatted_chat = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
+inputs = tokenizer(formatted_chat, return_tensors="pt", add_special_tokens=False)
+
+# Completion
+output = model.generate(inputs["input_ids"], max_new_tokens=100, temperature=0.5)
+print(tokenizer.decode(output[0][inputs['input_ids'].size(1):], skip_special_tokens=True))
+```
+
+**vLLM Serve Example**
+```bash
+vllm serve yulan-team/YuLan-Mini-Instruct --dtype bfloat16
+```
+
+**SGLang Serve Example**
+```bash
+python -m sglang.launch_server --model-path yulan-team/YuLan-Mini-Instruct --port 30000 --host 0.0.0.0
+```
+
+**Ollama**
+```bash
+ollama run hf.co/mradermacher/YuLan-Mini-Instruct-GGUF:IQ4_XS
+```
 
 ## Contributors
 
@@ -238,6 +329,12 @@ We propose YuLan-Mini-Instruct, a powerful small-scale language model with 2.4 b
 
 Authors are listed in alphabetical order: Fei Bai, Zhipeng Chen, Yanzipeng Gao, Yukai Gu, Yiwen Hu, Yihong Liu, Yingqian Min, Ruiyang Ren, Huatong Song, Shuang Sun, Ji-Rong Wen, Chenghao Wu, Xin Zhao, Kun Zhou, Yutao Zhu
 
+## License
+
+- The code in this repository, the model weights, and optimizer states are released under the [MIT License](./LICENSE).
+- Policies regarding the use of model weights, intermediate optimizer states, and training data will be announced in future updates.
+- Limitations: Despite our efforts to mitigate safety concerns and encourage the generation of ethical and lawful text, the probabilistic nature of language models may still lead to unexpected outputs. For instance, responses might contain bias, discrimination, or other harmful content. Please refrain from disseminating such content. We are not liable for any consequences arising from the spread of harmful information.
+
 ## Reference
 
 Please kindly cite our reports if they are helpful for your research.