Update README.md

README.md

@@ -41,11 +41,8 @@ Models can be accessed via TheStage AI Python SDK: ElasticModels, or deployed as
 
 ---
 
-
 ### System Requirements
 
----
-
 | **Property**| **Value** |
  | ---  | ---  |
 | **GPU** | L40s, RTX 5090, H100, B200 |
@@ -56,8 +53,6 @@ Models can be accessed via TheStage AI Python SDK: ElasticModels, or deployed as
 
 ### TheStage AI Access token setup
 
----
-
 Install TheStage AI CLI and setup API token:
 
 ```bash
@@ -67,8 +62,6 @@ thestage config set --access-token <YOUR_ACCESS_TOKEN>
 
 ### ElasticModels installation
 
----
-
 Install TheStage Elastic Models package:
 
 ```bash
@@ -81,7 +74,6 @@ pip install --force-reinstall --no-deps nvidia-cudnn-frontend==1.18.0
 
 ---
 
-
 Elastic Models provides the same interface as HuggingFace Transformers. Here is an example of how to use the Llama-3.1-8B-Instruct model:
 
 ```python
@@ -150,15 +142,12 @@ print(f"# A:\n{output}\n")
 
 ---
 
-
 We have used the `lm_eval` library to validate the models. For each model size (S, M, L, XL), we have run the following tasks: MMLU, PIQA, Arc Challenge, Winogrande.
 
 ![Quality Benchmarking](https://cdn.thestage.ai/production/cms_file_upload/1773422713-7d51617f-e70a-41db-95f9-abd0d9ff338f/Elastic_Llama_3.1_8B_Instruct_MMLU.png)
 
 ### Quality Benchmark Results
 
----
-
 | **Metric/Model Size**| **S**| **M**| **L**| **XL**| **Original**| **W8A8, int8** |
  | ---  | ---  | ---  | ---  | ---  | ---  | ---  |
 | **MMLU** | 67.4 | 68.1 | 68.3 | 68.5 | 68.4 | 24.3 |
@@ -171,7 +160,6 @@ We have used the `lm_eval` library to validate the models. For each model size (
 
 ---
 
-
 - **MMLU**: Measures model performance on a diverse set of multiple-choice questions covering various academic subjects, testing general knowledge and reasoning.
 - **PIQA**: Evaluates physical commonsense reasoning by asking the model to choose the most plausible solution to everyday physical problems.
 - **Arc Challenge**: Assesses scientific and factual reasoning using challenging multiple-choice questions from the AI2 Reasoning Challenge dataset.
@@ -181,7 +169,6 @@ We have used the `lm_eval` library to validate the models. For each model size (
 
 ---
 
-
 - **Accuracy**: Accuracy measures the proportion of model predictions that exactly match the correct answers across evaluation tasks.
 
 
@@ -189,15 +176,12 @@ We have used the `lm_eval` library to validate the models. For each model size (
 
 ---
 
-
 We measured TPS (tokens per second) for each model size using 100 input tokens and 300 output tokens.
 
 ![Latency Benchmarking](https://cdn.thestage.ai/production/cms_file_upload/1779128494-03458602-0144-446d-bebd-6d3ebc8b9907/Elastic_Llama_3.1_8B_Instruct_latency.png)
 
 ### Latency Benchmark Results
 
----
-
 Tokens per second for different model sizes on various GPUs.
 
 | **GPU/Model Size**| **S**| **M**| **L**| **XL**| **Original**| **W8A8_int8** |
@@ -213,7 +197,6 @@ Tokens per second for different model sizes on various GPUs.
 
 ---
 
-
 The benchmarking was performed on a single GPU with a batch size of 1. Each model was run for 10 iterations, and the average latency was calculated.
 
 > **Algorithm summary:**
@@ -233,19 +216,16 @@ The benchmarking was performed on a single GPU with a batch size of 1. Each mode
 
 ---
 
-
 For serving with Nvidia GPUs, we provide ready-to-go Docker containers with OpenAI-compatible API endpoints.
 Using our containers you can set up an inference endpoint on any desired cloud/serverless providers as well as on-premise servers.
 You can also use this container to run inference through TheStage AI platform.
 
 ### Prebuilt image from ECR
 
----
-
 Pull docker image and start inference container:
 
 ```bash
-docker pull public.ecr.aws/i3f7g5s7/thestage/elastic-models:0.2.0-llm-24.09c
+docker pull public.ecr.aws/i3f7g5s7/thestage/elastic-models:0.2.1.post0-llm-24.09a
 ```
 ```bash
 docker run --rm -ti \
@@ -258,7 +238,7 @@ docker run --rm -ti \
   -e HUGGINGFACE_ACCESS_TOKEN=<HUGGINGFACE_ACCESS_TOKEN> \
   -e THESTAGE_AUTH_TOKEN=<THESTAGE_ACCESS_TOKEN> \
   -v /mnt/hf_cache:/root/.cache/huggingface \
-  public.ecr.aws/i3f7g5s7/thestage/elastic-models:0.2.0-llm-24.09c
+  public.ecr.aws/i3f7g5s7/thestage/elastic-models:0.2.1.post0-llm-24.09a
 ```
 
 | **Parameter**              | **Description**                                                                                      |
@@ -273,7 +253,6 @@ docker run --rm -ti \
 
 ---
 
-
 You can invoke the endpoint using CURL as follows:
 
 ```bash
@@ -316,17 +295,12 @@ print(response.choices[0].message.content)
 
 ---
 
-
 ### Method
 
----
-
 > **POST** `/v1/chat/completions`
 
 ### Header Parameters
 
----
-
 > `Authorization`: `string`
 >
 > Bearer token for authentication. Should match the `AUTH_TOKEN` set during container startup.
@@ -341,8 +315,6 @@ print(response.choices[0].message.content)
 
 ### Input Body
 
----
-
 > `messages` : `string`
 >
 > The input text prompt.
@@ -352,13 +324,10 @@ print(response.choices[0].message.content)
 
 ---
 
-
 For more details please use the tutorial [Modal deployment](https://docs.thestage.ai/tutorials/source/modal_thestage.html)
 
 ### Clone modal serving code
 
----
-
 ```shell
 git clone https://github.com/TheStageAI/ElasticModels.git
 cd ElasticModels/examples/modal
@@ -366,8 +335,6 @@ cd ElasticModels/examples/modal
 
 ### Configuration of environment variables
 
----
-
 Set your environment variables in `modal_serving.py`:
 
 ```python
@@ -386,8 +353,6 @@ ENVS = {
 
 ### Configuration of GPUs
 
----
-
 Set your desired GPU type and autoscaling variables in `modal_serving.py`:
 
 ```python
@@ -414,8 +379,6 @@ def serve():
 
 ### Run serving
 
----
-
 ```shell
 modal serve modal_serving.py
 ```