Update README.md
Browse files
README.md
CHANGED
|
@@ -14,23 +14,13 @@ base_model: QCRI/Fanar-1-9B-Instruct
|
|
| 14 |
|
| 15 |
# Fanar-1-9B-Instruct-AWQ
|
| 16 |
|
| 17 |
-
|
| 18 |
|
| 19 |
-
## Quick Start
|
| 20 |
|
| 21 |
-
```python
|
| 22 |
-
from vllm import LLM
|
| 23 |
-
|
| 24 |
-
llm = LLM(
|
| 25 |
-
model="buthainaaa/Fanar-1-9B-Instruct-GPTQ",
|
| 26 |
-
quantization="awq",
|
| 27 |
-
dtype="half"
|
| 28 |
-
)
|
| 29 |
-
```
|
| 30 |
|
| 31 |
## Details
|
| 32 |
|
| 33 |
-
- **Quantization:**
|
| 34 |
- **Size:** ~5GB (vs ~18GB original)
|
| 35 |
- **Memory:** 75% reduction
|
| 36 |
- **Quality:** 95%+ retention
|
|
|
|
| 14 |
|
| 15 |
# Fanar-1-9B-Instruct-AWQ
|
| 16 |
|
| 17 |
+
GPTQ 4-bit quantized version of [QCRI/Fanar-1-9B-Instruct](https://huggingface.co/QCRI/Fanar-1-9B-Instruct).
|
| 18 |
|
|
|
|
| 19 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
## Details
|
| 22 |
|
| 23 |
+
- **Quantization:** GPTQ 4-bit (w4a16)
|
| 24 |
- **Size:** ~5GB (vs ~18GB original)
|
| 25 |
- **Memory:** 75% reduction
|
| 26 |
- **Quality:** 95%+ retention
|