Update README.md

README.md

@@ -13,15 +13,12 @@ tags:
 
 We present CrystalChat, an instruction following model finetuned from [LLM360/CrystalCoder](https://huggingface.co/LLM360/CrystalCoder).
 
-| Model | Trained Tokens | ARC | HellaSwag | MMLU (5-shot) | TruthfulQA | Language Avg. | HumanEval (pass@1) | MBPP (pass@1) | Coding Avg. | Avg. of Avg.|
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | 
-| Mistral 7B | - | 59.98 | 83.31 | 64.16 | 42.15 | 62.40 | 29.12 | 38.78 | 33.95 | 48.68 |
-| **CrystalChat 7B** | 1.4T | 51.71 | 76.12 | 53.22 | 47.29 | 57.08 | 34.12 | 39.11 | 36.62 | 46.85 |
-| CrystalCoder 7B | 1.4T | 47.01 | 71.97 | 48.78 | 35.91 | 50.92 | 28.38 | 36.38 | 32.38 | 41.65 |
-| CodeLlaMA 7B | 2.5T | 39.93 | 60.80 | 31.12 | 37.82 | 42.42 | 33.50 | 41.40 | 37.45 | 39.94 |
-| OpenLLaMA v2 7B | 1T | 43.60 | 72.20 | 41.29 | 35.54 | 48.18 | 15.32 | 12.69 | 28.01 | 38.10 | 
-| LLaMA 2 7B | 2T | 53.07 | 77.74 | 43.80 | 38.98 | 53.39 | 13.05 | 20.09 | 16.57 | 34.98 |
-| StarCoder-15B | 1.03 | - | - | - | - | - | 33.63 | 43.28 | 38.46 | - | 
+| Model | Trained Tokens | ARC | HellaSwag | MMLU (5-shot) | GSM8K | Winogrande(5-shot) | TruthfulQA | Language Avg. | HumanEval (pass@1) | MBPP (pass@1) | Coding Avg. | Avg. of Avg.|
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| Mistral-7B-Instruct-v0.1 | - | 54.86 | 75.71 | 55.56 | 32.00 | 74.27 | 55.90 | 58.05 | 29.27 | 31.96 | 30.62 | 44.34 |
+| **CrystalChat 7B** | 1.4T | 51.71 | 76.12 | 53.22 | 28.05 | 70.64 | 47.29 | 53.29 | 34.12 | 39.11 | 36.62 | 50.07 |
+| CodeLlama-7b-Instruct | 2.5T | 43.35 | 66.14 | 42.75 | 15.92 | 64.33 | 39.23 | 45.29 | 34.12 | 38.91 | 36.52 | 40.91 |
+| Llama-2-7b-Chat | 2T | 53.07 | 78.39 | 48.42 | 18.88 | 73.09 | 45.30 | 52.86 | 13.26 | 17.43 | 15.35 | 34.11 |
 
 ## Model Description
 
@@ -38,14 +35,15 @@ We present CrystalChat, an instruction following model finetuned from [LLM360/Cr
 
 ```python
 import torch
-from transformers import LlamaTokenizer, LlamaForCausalLM
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-tokenizer = LlamaTokenizer.from_pretrained("LLM360/CrystalChat/", trust_remote_code=True)
-model = LlamaForCausalLM.from_pretrained("LLM360/CrystalChat", trust_remote_code=True)
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+tokenizer = AutoTokenizer.from_pretrained("LLM360/CrystalChat", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("LLM360/CrystalChat", trust_remote_code=True).to(device)
 
 prompt = 'int add(int x, int y) {'
 
-input_ids = tokenizer(prompt, return_tensors="pt").input_ids
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
 gen_tokens = model.generate(input_ids, do_sample=True, max_length=400)
 
 print("-"*20 + "Output for model"  + 20 * '-')