Update quantization notes for pure v1
Browse files- QUANTIZATION.md +3 -3
QUANTIZATION.md
CHANGED
|
@@ -17,7 +17,7 @@ ONNX Runtime `quantize_dynamic` with uint8 weights, per-channel, reduce_range, a
|
|
| 17 |
| `n g a` | 0.820 | 0.500 |
|
| 18 |
| `pot` | 0.331 | 0.714 |
|
| 19 |
|
| 20 |
-
On the v44 model, benchmark catch fell from 93.9% to about 84.1%. The current
|
| 21 |
|
| 22 |
### Hybrid, fp16 embedding plus INT8 layers, 482 MB
|
| 23 |
|
|
@@ -35,7 +35,7 @@ Q8BERT showed that 8-bit quantization-aware training keeps BERT within about 1%
|
|
| 35 |
|
| 36 |
QDQBERT is the same idea in the Hugging Face and TensorRT toolchain. It inserts QuantizeLinear and DequantizeLinear pairs for linear inputs and weights, MatMul inputs and residual adds, then runs QAT. The published recipe uses percentile calibration at 99.99 for BERT style models.
|
| 37 |
|
| 38 |
-
This is the highest confidence route for send-gate: start from the fp32
|
| 39 |
|
| 40 |
### Static S8S8 QDQ is the first thing to retry
|
| 41 |
|
|
@@ -120,7 +120,7 @@ Keep the embedding in fp16 or 8-bit PEG, keep the sensitive residual and FFN ten
|
|
| 120 |
|
| 121 |
### Phase 3: QAT W8A8
|
| 122 |
|
| 123 |
-
If Phase 1 and Phase 2 miss the target, run QAT from the fp32
|
| 124 |
|
| 125 |
### Phase 4: only then try 4-bit
|
| 126 |
|
|
|
|
| 17 |
| `n g a` | 0.820 | 0.500 |
|
| 18 |
| `pot` | 0.331 | 0.714 |
|
| 19 |
|
| 20 |
+
On the v44 model, benchmark catch fell from 93.9% to about 84.1%. The current pure v1 fp32 model scores 92.9% with no routing and no rules. Short phrases moved the most in both cases. The model did not simply become less confident; some benign short phrases became more confident.
|
| 21 |
|
| 22 |
### Hybrid, fp16 embedding plus INT8 layers, 482 MB
|
| 23 |
|
|
|
|
| 35 |
|
| 36 |
QDQBERT is the same idea in the Hugging Face and TensorRT toolchain. It inserts QuantizeLinear and DequantizeLinear pairs for linear inputs and weights, MatMul inputs and residual adds, then runs QAT. The published recipe uses percentile calibration at 99.99 for BERT style models.
|
| 37 |
|
| 38 |
+
This is the highest confidence route for send-gate: start from the fp32 pure v1 checkpoint, add fake quantization, distill from the fp32 model, and evaluate decision parity.
|
| 39 |
|
| 40 |
### Static S8S8 QDQ is the first thing to retry
|
| 41 |
|
|
|
|
| 120 |
|
| 121 |
### Phase 3: QAT W8A8
|
| 122 |
|
| 123 |
+
If Phase 1 and Phase 2 miss the target, run QAT from the fp32 pure v1 checkpoint. Use QDQ fake quantization on MatMul and linear weights and activations, keep LayerNorm, GELU and softmax in fp32, and distill from the fp32 pure v1 logits. Use the same training data and the same threshold sweep. Two thousand steps is cheap for a 307M encoder on the current GPU.
|
| 124 |
|
| 125 |
### Phase 4: only then try 4-bit
|
| 126 |
|