Update README.md
Browse files
README.md
CHANGED
|
@@ -27,7 +27,7 @@ This model was obtained by quantizing the weights of [Llama-2-7b-chat](https://h
|
|
| 27 |
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
|
| 28 |
|
| 29 |
Only the weights of the linear operators within transformers blocks are quantized. Symmetric per-channel quantization is applied, in which a linear scaling per output dimension maps the INT8 and floating point representations of the quantized weights.
|
| 30 |
-
[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) is used for quantization with 1% damping factor and 256 sequences of
|
| 31 |
|
| 32 |
|
| 33 |
## Deployment
|
|
|
|
| 27 |
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
|
| 28 |
|
| 29 |
Only the weights of the linear operators within transformers blocks are quantized. Symmetric per-channel quantization is applied, in which a linear scaling per output dimension maps the INT8 and floating point representations of the quantized weights.
|
| 30 |
+
[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) is used for quantization with 1% damping factor and 256 sequences of 4,096 random tokens.
|
| 31 |
|
| 32 |
|
| 33 |
## Deployment
|