yamura4 commited on
Commit
e33abcd
·
verified ·
1 Parent(s): a096af9

Update quantization notes for pure v1

Browse files
Files changed (1) hide show
  1. QUANTIZATION.md +3 -3
QUANTIZATION.md CHANGED
@@ -17,7 +17,7 @@ ONNX Runtime `quantize_dynamic` with uint8 weights, per-channel, reduce_range, a
17
  | `n g a` | 0.820 | 0.500 |
18
  | `pot` | 0.331 | 0.714 |
19
 
20
- On the v44 model, benchmark catch fell from 93.9% to about 84.1%. The current v54 fp32 model scores 94.2% with model-only routing. Short phrases moved the most in both cases. The model did not simply become less confident; some benign short phrases became more confident.
21
 
22
  ### Hybrid, fp16 embedding plus INT8 layers, 482 MB
23
 
@@ -35,7 +35,7 @@ Q8BERT showed that 8-bit quantization-aware training keeps BERT within about 1%
35
 
36
  QDQBERT is the same idea in the Hugging Face and TensorRT toolchain. It inserts QuantizeLinear and DequantizeLinear pairs for linear inputs and weights, MatMul inputs and residual adds, then runs QAT. The published recipe uses percentile calibration at 99.99 for BERT style models.
37
 
38
- This is the highest confidence route for send-gate: start from the fp32 v54 checkpoint, add fake quantization, distill from the fp32 model, and evaluate decision parity.
39
 
40
  ### Static S8S8 QDQ is the first thing to retry
41
 
@@ -120,7 +120,7 @@ Keep the embedding in fp16 or 8-bit PEG, keep the sensitive residual and FFN ten
120
 
121
  ### Phase 3: QAT W8A8
122
 
123
- If Phase 1 and Phase 2 miss the target, run QAT from the fp32 v54 checkpoint. Use QDQ fake quantization on MatMul and linear weights and activations, keep LayerNorm, GELU and softmax in fp32, and distill from the fp32 v54 logits. Use the same training data and the same threshold sweep. Two thousand steps is cheap for a 307M encoder on the current GPU.
124
 
125
  ### Phase 4: only then try 4-bit
126
 
 
17
  | `n g a` | 0.820 | 0.500 |
18
  | `pot` | 0.331 | 0.714 |
19
 
20
+ On the v44 model, benchmark catch fell from 93.9% to about 84.1%. The current pure v1 fp32 model scores 92.9% with no routing and no rules. Short phrases moved the most in both cases. The model did not simply become less confident; some benign short phrases became more confident.
21
 
22
  ### Hybrid, fp16 embedding plus INT8 layers, 482 MB
23
 
 
35
 
36
  QDQBERT is the same idea in the Hugging Face and TensorRT toolchain. It inserts QuantizeLinear and DequantizeLinear pairs for linear inputs and weights, MatMul inputs and residual adds, then runs QAT. The published recipe uses percentile calibration at 99.99 for BERT style models.
37
 
38
+ This is the highest confidence route for send-gate: start from the fp32 pure v1 checkpoint, add fake quantization, distill from the fp32 model, and evaluate decision parity.
39
 
40
  ### Static S8S8 QDQ is the first thing to retry
41
 
 
120
 
121
  ### Phase 3: QAT W8A8
122
 
123
+ If Phase 1 and Phase 2 miss the target, run QAT from the fp32 pure v1 checkpoint. Use QDQ fake quantization on MatMul and linear weights and activations, keep LayerNorm, GELU and softmax in fp32, and distill from the fp32 pure v1 logits. Use the same training data and the same threshold sweep. Two thousand steps is cheap for a 307M encoder on the current GPU.
124
 
125
  ### Phase 4: only then try 4-bit
126