Update README.md

README.md

@@ -3,11 +3,64 @@ license: other
 license_name: iquestcoder
 license_link: https://huggingface.co/IQuestLab/IQuest-Coder-V1-40B-Instruct/blob/main/LICENSE
 language:
-  - en
+- en
 library_name: transformers
 base_model: IQuestLab/IQuest-Coder-V1-40B-Instruct
 ---
 
+# IQuest-Coder-V1-40B-Instruct AWQ - INT8
+
+## Model Details
+
+### Quantization Details
+
+- **Quantization Method:** AWQ
+- **Bits:** 8
+- **Group Size:** 32
+- **Calibration Dataset:** [princeton-nlp/SWE-bench_Verified](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified)
+- **Quantization Tool:** [llm-compressor](https://github.com/vllm-project/llm-compressor)
+
+### Memory Usage
+
+| **Type** | **IQuest-Coder-V1-40B-Instruct** | **IQuest-Coder-V1-40B-Instruct-AWQ-8bit** |
+|:---------------:|:----------------:|:----------------:|
+| **Memory Size** | 74.1 GB | 40.1 GB | 
+| **KV Cache per Token** | 320.0 kB | 160.0 kB | 
+| **KV Cache per Context** | 40.0 GB | 20.0 GB | 
+
+## Inference
+
+### Prerequisite
+
+
+```bash 
+git clone https://github.com/vllm-project/vllm.git
+cd vllm
+git fetch origin pull/31575/head:pr-31575
+git checkout pr-31575
+VLLM_USE_PRECOMPILED=1 pip install -e .
+```
+
+
+### Basic Usage
+
+
+```bash 
+vllm serve cyankiwi/IQuest-Coder-V1-40B-Instruct-AWQ-8bit --trust-remote-code
+```
+
+
+## Additional Information
+
+### Changelog
+
+- **v1.0.0** - Initial quantized release
+
+### Authors
+
+- **Name:** Ton Cao
+- **Contacts:** ton@cyan.kiwi
+
 
 ![Evaluation Results](./papers/iquest-coder-v1-logo.png)