TheStageAI
/

Elastic-Llama-3.1-8B-Instruct

Text Generation

Model card Files Files and versions

Update README.md

#21

by hinairo - opened May 28

base: refs/heads/main

←

from: refs/pr/21

Discussion Files changed

Files changed (1) hide show

README.md +5 -5

README.md CHANGED Viewed

@@ -192,7 +192,7 @@ We have used the `lm_eval` library to validate the models. For each model size (
 We measured TPS (tokens per second) for each model size using 100 input tokens and 300 output tokens.
-![Latency Benchmarking](https://cdn.thestage.ai/production/cms_file_upload/1773422728-414fdb22-5c04-44a6-8686-0602a4293e88/Elastic_Llama_3.1_8B_Instruct_latency.png)
 ### Latency Benchmark Results
@@ -202,10 +202,10 @@ Tokens per second for different model sizes on various GPUs.
 | **GPU/Model Size**| **S**| **M**| **L**| **XL**| **Original**| **W8A8_int8** |
  | ---  | ---  | ---  | ---  | ---  | ---  | ---  |
-| **H100** | 189 | 168 | 156 | 134 | 60 | 191 |
-| **L40s** | 72 | 63 | 56 | 45 | 37 | 77 |
-| **B200** | 239 | 236 | 207 | 199 | 100 | N/A |
-| **GeForce RTX 5090** | 143 | N/A | N/A | N/A | 60 | N/A |
 | **GeForce RTX 4090** | 95 | N/A | N/A | N/A | 41 | N/A |

 We measured TPS (tokens per second) for each model size using 100 input tokens and 300 output tokens.
+![Latency Benchmarking](https://cdn.thestage.ai/production/cms_file_upload/1779128494-03458602-0144-446d-bebd-6d3ebc8b9907/Elastic_Llama_3.1_8B_Instruct_latency.png)
 ### Latency Benchmark Results
 | **GPU/Model Size**| **S**| **M**| **L**| **XL**| **Original**| **W8A8_int8** |
  | ---  | ---  | ---  | ---  | ---  | ---  | ---  |
+| **H100** | 204 | 182 | 168 | 143 | 60 | 191 |
+| **L40s** | 73 | 64 | 57 | 46 | 37 | 77 |
+| **B200** | 276 | 270 | 264 | 249 | 100 | N/A |
+| **GeForce RTX 5090** | 153 | N/A | N/A | N/A | 60 | N/A |
 | **GeForce RTX 4090** | 95 | N/A | N/A | N/A | 41 | N/A |