mlboydaisuke commited on
Commit
cab8e6b
·
verified ·
1 Parent(s): 3093496

head-quant docs: per-block-32 absmax ship shape (per-channel = beta delegate bug; naming note)

Browse files
Files changed (1) hide show
  1. README.md +9 -3
README.md CHANGED
@@ -29,12 +29,18 @@ token-identical to the Mac GPU** on both fixed prompts.
29
  ## Bundles
30
 
31
  - **`gpu-pipelined/qwen3_5_2b_decode_int8hu_perchan_sym/` — the ship config (2.9 GB)**:
32
- transformer int8 linear per-block-32 + **untied lm_head in per-channel absmax int8**
33
- (`int8hu --head-quant perchan --head-sym`). The head trick is what unlocks the speed: the
34
  248 K-vocab fp16 head was ~1.0 GB of the ~2.4 GB per-token read. Crucial detail: the head
35
  must be quantized with plain **absmax `symmetric`** — the default
36
  `symmetric_with_clipping` clips outlier head rows and flips oracle top-1s (full story in
37
  the zoo's [pipelined-engine notes](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)).
 
 
 
 
 
 
38
  - `gpu-pipelined/qwen3_5_2b_decode_int8lin/` — fp16-head variant (2.4 GB): 127 tok/s Mac /
39
  19–21 iPhone. Smaller; keep if you want the head at full precision.
40
 
@@ -78,5 +84,5 @@ The ship bundle decodes 28–30 tok/s on iPhone 17 Pro with exact numerics. Know
78
 
79
  Conversion script (self-contained) + method page in the zoo:
80
  [`conversion/export_qwen3_5_decode_pipelined.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_qwen3_5_decode_pipelined.py)
81
- (`int8hu --head-quant perchan --head-sym --hf-id Qwen/Qwen3.5-2B`) ·
82
  [`knowledge/pipelined-engine.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)
 
29
  ## Bundles
30
 
31
  - **`gpu-pipelined/qwen3_5_2b_decode_int8hu_perchan_sym/` — the ship config (2.9 GB)**:
32
+ transformer int8 linear per-block-32 + **untied lm_head in per-block-32 absmax int8**
33
+ (`int8hu --head-sym`). The head trick is what unlocks the speed: the
34
  248 K-vocab fp16 head was ~1.0 GB of the ~2.4 GB per-token read. Crucial detail: the head
35
  must be quantized with plain **absmax `symmetric`** — the default
36
  `symmetric_with_clipping` clips outlier head rows and flips oracle top-1s (full story in
37
  the zoo's [pipelined-engine notes](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)).
38
+ **Naming note (2026-06-11):** the directory says `_perchan_sym`, but its head is
39
+ per-block-32 — the export script of the day parsed the granularity flag without applying
40
+ it (since fixed); byte-identical bundle sizes confirmed it. All numbers were measured on
41
+ exactly these bytes and stand. Genuinely per-channel (axis-0) int8 weights are **broken
42
+ on the current beta GPU delegate** (garbage logits), so per-block-32 + `symmetric` IS the
43
+ correct ship shape. The dir name is kept to avoid breaking download paths.
44
  - `gpu-pipelined/qwen3_5_2b_decode_int8lin/` — fp16-head variant (2.4 GB): 127 tok/s Mac /
45
  19–21 iPhone. Smaller; keep if you want the head at full precision.
46
 
 
84
 
85
  Conversion script (self-contained) + method page in the zoo:
86
  [`conversion/export_qwen3_5_decode_pipelined.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_qwen3_5_decode_pipelined.py)
87
+ (`int8hu --head-sym --hf-id Qwen/Qwen3.5-2B`) ·
88
  [`knowledge/pipelined-engine.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)