Add detailed quantization environment to model card
#2
by larryli2 - opened
README.md
CHANGED
|
@@ -52,6 +52,22 @@ python3 quantize_quark.py \
|
|
| 52 |
|
| 53 |
No calibration dataset is required for this file-to-file quantization path.
|
| 54 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
### vLLM Loading Note
|
| 56 |
|
| 57 |
When using this FP8 Eagle3 checkpoint as a vLLM draft model, make sure the exported `config.json` records the excluded layers as regex patterns. If Quark exports:
|
|
|
|
| 52 |
|
| 53 |
No calibration dataset is required for this file-to-file quantization path.
|
| 54 |
|
| 55 |
+
### Quantization Environment
|
| 56 |
+
|
| 57 |
+
Quantization was run on a single AMD GPU (ROCm) using the following software stack:
|
| 58 |
+
|
| 59 |
+
| Component | Version |
|
| 60 |
+
| --- | --- |
|
| 61 |
+
| OS | Linux (x86_64) |
|
| 62 |
+
| Python | 3.11 |
|
| 63 |
+
| PyTorch | 2.9.0 (ROCm 6.4 build) |
|
| 64 |
+
| `pytorch-triton-rocm` | 3.5.0 |
|
| 65 |
+
| Transformers | 4.57.1 |
|
| 66 |
+
| `huggingface_hub` | 0.36.0 |
|
| 67 |
+
| `accelerate` | 1.11.0 |
|
| 68 |
+
| `safetensors` | 0.6.2 |
|
| 69 |
+
| `datasets` | 3.6.0 |
|
| 70 |
+
|
| 71 |
### vLLM Loading Note
|
| 72 |
|
| 73 |
When using this FP8 Eagle3 checkpoint as a vLLM draft model, make sure the exported `config.json` records the excluded layers as regex patterns. If Quark exports:
|