Update README.md

README.md

@@ -42,7 +42,7 @@ from transformers import AutoTokenizer
 from vllm import LLM, SamplingParams
 
 max_model_len, tp_size = 4096, 1
-model_name = "neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic"
+model_name = "neuralmagic/granite-3.1-8b-instruct-FP8-dynamic"
 tokenizer = AutoTokenizer.from_pretrained(model_name)
 llm = LLM(model=model_name, tensor_parallel_size=tp_size, max_model_len=max_model_len, trust_remote_code=True)
 sampling_params = SamplingParams(temperature=0.3, max_tokens=256, stop_token_ids=[tokenizer.eos_token_id])
@@ -65,6 +65,8 @@ vLLM also supports OpenAI-compatible serving. See the [documentation](https://do
 
 This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below. 
 
+<details>
+  <summary>Model Creation Code</summary>
 
 ```bash
 python quantize.py --model_id ibm-granite/granite-3.1-8b-instruct --save_path "output_dir/"
@@ -110,16 +112,20 @@ def main():
 if __name__ == "__main__":
     main()
 ```
+</details>
 
 ## Evaluation
 
-The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
+The model was evaluated on OpenLLM Leaderboard [V1](https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard), OpenLLM Leaderboard [V2](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/) and on [HumanEval](https://github.com/neuralmagic/evalplus), using the following commands:
 
+<details>
+<summary>Evaluation Commands</summary>
+  
 OpenLLM Leaderboard V1:
 ```
 lm_eval \
   --model vllm \
-  --model_args pretrained="neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --model_args pretrained="neuralmagic/granite-3.1-8b-instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
   --tasks openllm \
   --write_out \
   --batch_size auto \
@@ -127,11 +133,23 @@ lm_eval \
   --show_config
 ```
 
+OpenLLM Leaderboard V2:
+```
+lm_eval \
+  --model vllm \
+  --model_args pretrained="neuralmagic/granite-3.1-8b-instruct-FP8-dynamic",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=1,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
+  --tasks leaderboard \
+  --write_out \
+  --batch_size auto \
+  --output_path output_dir \
+  --show_config
+```
+
 #### HumanEval
 ##### Generation
 ```
 python3 codegen/generate.py \
-  --model neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic \
+  --model neuralmagic/granite-3.1-8b-instruct-FP8-dynamic \
   --bs 16 \
   --temperature 0.2 \
   --n_samples 50 \
@@ -141,45 +159,128 @@ python3 codegen/generate.py \
 ##### Sanitization
 ```
 python3 evalplus/sanitize.py \
-  humaneval/neuralmagic-ent--granite-3.1-8b-instruct-FP8-dynamic_vllm_temp_0.2
+  humaneval/neuralmagic--granite-3.1-8b-instruct-FP8-dynamic_vllm_temp_0.2
 ```
 ##### Evaluation
 ```
 evalplus.evaluate \
   --dataset humaneval \
-  --samples humaneval/neuralmagic-ent--granite-3.1-8b-instruct-FP8-dynamic_vllm_temp_0.2-sanitized
+  --samples humaneval/neuralmagic--granite-3.1-8b-instruct-FP8-dynamic_vllm_temp_0.2-sanitized
 ```
+</details>
 
 ### Accuracy
 
-#### OpenLLM Leaderboard V1 evaluation scores
-
-| Metric                                  | ibm-granite/granite-3.1-8b-instruct             | neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| ARC-Challenge (Acc-Norm, 25-shot)       | 66.81                             | 66.81                                      |
-| GSM8K (Strict-Match, 5-shot)            | 64.52                             | 66.64                                       |
-| HellaSwag (Acc-Norm, 10-shot)           | 84.18                             | 84.16                                       |
-| MMLU (Acc, 5-shot)                      | 65.52                             | 65.36                                        |
-| TruthfulQA (MC2, 0-shot)                | 60.57                             | 60.52                                        |
-| Winogrande (Acc, 5-shot)                | 80.19                             | 79.95                                        |
-| **Average Score**                       | **70.30**                         | **70.57**                                   |
-| **Recovery**                            | **100.00**                        | **100.39**                                   |
-
-| Metric                                  | ibm-granite/granite-3.1-8b-instruct             | neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| IFEval (Inst Level Strict Acc, 0-shot)| 74.10                           | 73.62                                         |
-| BBH (Acc-Norm, 3-shot)            | 53.19                             | 53.26                                         |
-| Math-Hard (Exact-Match, 4-shot)   | 14.77                            | 16.79                                       |
-| GPQA (Acc-Norm, 0-shot)           | 31.76                             | 32.58                                        |
-| MUSR (Acc-Norm, 0-shot)           | 46.01                             | 47.34                                        |
-| MMLU-Pro (Acc, 5-shot)            | 35.81                             | 35.72                                        |
-| **Average Score**                 | **42.61**                         | **43.22**                                    |
-| **Recovery**                      | **100.00**                         | **101.43**                                    |
-
-#### HumanEval pass@1 scores
-| Metric                                  | ibm-granite/granite-3.1-8b-instruct             | neuralmagic-ent/granite-3.1-8b-instruct-FP8-dynamic |
-|-----------------------------------------|:---------------------------------:|:-------------------------------------------:|
-| HumanEval Pass@1                        | 71.00                             |  69.90                                     |
+<table>
+  <thead>
+    <tr>
+      <th>Category</th>
+      <th>Metric</th>
+      <th>ibm-granite/granite-3.1-8b-instruct</th>
+      <th>neuralmagic/granite-3.1-8b-instruct-FP8-dynamic</th>
+      <th>Recovery (%)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <!-- OpenLLM Leaderboard V1 -->
+    <tr>
+      <td rowspan="7"><b>OpenLLM Leaderboard V1</b></td>
+      <td>ARC-Challenge (Acc-Norm, 25-shot)</td>
+      <td>66.81</td>
+      <td>66.81</td>
+      <td>100.00</td>
+    </tr>
+    <tr>
+      <td>GSM8K (Strict-Match, 5-shot)</td>
+      <td>64.52</td>
+      <td>66.64</td>
+      <td>103.29</td>
+    </tr>
+    <tr>
+      <td>HellaSwag (Acc-Norm, 10-shot)</td>
+      <td>84.18</td>
+      <td>84.16</td>
+      <td>99.98</td>
+    </tr>
+    <tr>
+      <td>MMLU (Acc, 5-shot)</td>
+      <td>65.52</td>
+      <td>65.36</td>
+      <td>99.76</td>
+    </tr>
+    <tr>
+      <td>TruthfulQA (MC2, 0-shot)</td>
+      <td>60.57</td>
+      <td>60.52</td>
+      <td>99.92</td>
+    </tr>
+    <tr>
+      <td>Winogrande (Acc, 5-shot)</td>
+      <td>80.19</td>
+      <td>79.95</td>
+      <td>99.70</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>70.30</b></td>
+      <td><b>70.57</b></td>
+      <td><b>100.39</b></td>
+    </tr>
+    <!-- OpenLLM Leaderboard V2 -->
+    <tr>
+      <td rowspan="7"><b>OpenLLM Leaderboard V2</b></td>
+      <td>IFEval (Inst Level Strict Acc, 0-shot)</td>
+      <td>74.10</td>
+      <td>73.62</td>
+      <td>99.35</td>
+    </tr>
+    <tr>
+      <td>BBH (Acc-Norm, 3-shot)</td>
+      <td>53.19</td>
+      <td>53.26</td>
+      <td>100.13</td>
+    </tr>
+    <tr>
+      <td>Math-Hard (Exact-Match, 4-shot)</td>
+      <td>14.77</td>
+      <td>16.79</td>
+      <td>113.66</td>
+    </tr>
+    <tr>
+      <td>GPQA (Acc-Norm, 0-shot)</td>
+      <td>31.76</td>
+      <td>32.58</td>
+      <td>102.58</td>
+    </tr>
+    <tr>
+      <td>MUSR (Acc-Norm, 0-shot)</td>
+      <td>46.01</td>
+      <td>47.34</td>
+      <td>102.89</td>
+    </tr>
+    <tr>
+      <td>MMLU-Pro (Acc, 5-shot)</td>
+      <td>35.81</td>
+      <td>35.72</td>
+      <td>99.75</td>
+    </tr>
+    <tr>
+      <td><b>Average Score</b></td>
+      <td><b>42.61</b></td>
+      <td><b>43.22</b></td>
+      <td><b>101.43</b></td>
+    </tr>
+    <!-- HumanEval -->
+    <tr>
+      <td rowspan="2"><b>HumanEval</b></td>
+      <td>HumanEval Pass@1</td>
+      <td>71.00</td>
+      <td>69.90</td>
+      <td><b>98.45</b></td>
+    </tr>
+  </tbody>
+</table>
+
 
 
 ## Inference Performance
@@ -188,6 +289,16 @@ evalplus.evaluate \
 This model achieves up to 1.5x speedup in single-stream deployment and up to 1.1x speedup in multi-stream asynchronous deployment on L40 GPUs.
 The following performance benchmarks were conducted with [vLLM](https://docs.vllm.ai/en/latest/) version 0.6.6.post1, and [GuideLLM](https://github.com/neuralmagic/guidellm).
 
+<details>
+<summary>Benchmarking Command</summary>
+
+```
+guidellm --model neuralmagic/granite-3.1-8b-instruct-FP8-dynamic --target "http://localhost:8000/v1" --data-type emulated --data "prompt_tokens=<prompt_tokens>,generated_tokens=<generated_tokens>" --max seconds 360 --backend aiohttp_server
+```
+
+</details>
+
+
 ### Single-stream performance (measured with vLLM version 0.6.6.post1)
 <table>
   <tr>