head-quant docs: per-block-32 absmax ship shape (per-channel = beta delegate bug; naming note)
Browse files
README.md
CHANGED
|
@@ -29,12 +29,18 @@ token-identical to the Mac GPU** on both fixed prompts.
|
|
| 29 |
## Bundles
|
| 30 |
|
| 31 |
- **`gpu-pipelined/qwen3_5_2b_decode_int8hu_perchan_sym/` — the ship config (2.9 GB)**:
|
| 32 |
-
transformer int8 linear per-block-32 + **untied lm_head in per-
|
| 33 |
-
(`int8hu --head-
|
| 34 |
248 K-vocab fp16 head was ~1.0 GB of the ~2.4 GB per-token read. Crucial detail: the head
|
| 35 |
must be quantized with plain **absmax `symmetric`** — the default
|
| 36 |
`symmetric_with_clipping` clips outlier head rows and flips oracle top-1s (full story in
|
| 37 |
the zoo's [pipelined-engine notes](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
- `gpu-pipelined/qwen3_5_2b_decode_int8lin/` — fp16-head variant (2.4 GB): 127 tok/s Mac /
|
| 39 |
19–21 iPhone. Smaller; keep if you want the head at full precision.
|
| 40 |
|
|
@@ -78,5 +84,5 @@ The ship bundle decodes 28–30 tok/s on iPhone 17 Pro with exact numerics. Know
|
|
| 78 |
|
| 79 |
Conversion script (self-contained) + method page in the zoo:
|
| 80 |
[`conversion/export_qwen3_5_decode_pipelined.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_qwen3_5_decode_pipelined.py)
|
| 81 |
-
(`int8hu --head-
|
| 82 |
[`knowledge/pipelined-engine.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)
|
|
|
|
| 29 |
## Bundles
|
| 30 |
|
| 31 |
- **`gpu-pipelined/qwen3_5_2b_decode_int8hu_perchan_sym/` — the ship config (2.9 GB)**:
|
| 32 |
+
transformer int8 linear per-block-32 + **untied lm_head in per-block-32 absmax int8**
|
| 33 |
+
(`int8hu --head-sym`). The head trick is what unlocks the speed: the
|
| 34 |
248 K-vocab fp16 head was ~1.0 GB of the ~2.4 GB per-token read. Crucial detail: the head
|
| 35 |
must be quantized with plain **absmax `symmetric`** — the default
|
| 36 |
`symmetric_with_clipping` clips outlier head rows and flips oracle top-1s (full story in
|
| 37 |
the zoo's [pipelined-engine notes](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)).
|
| 38 |
+
**Naming note (2026-06-11):** the directory says `_perchan_sym`, but its head is
|
| 39 |
+
per-block-32 — the export script of the day parsed the granularity flag without applying
|
| 40 |
+
it (since fixed); byte-identical bundle sizes confirmed it. All numbers were measured on
|
| 41 |
+
exactly these bytes and stand. Genuinely per-channel (axis-0) int8 weights are **broken
|
| 42 |
+
on the current beta GPU delegate** (garbage logits), so per-block-32 + `symmetric` IS the
|
| 43 |
+
correct ship shape. The dir name is kept to avoid breaking download paths.
|
| 44 |
- `gpu-pipelined/qwen3_5_2b_decode_int8lin/` — fp16-head variant (2.4 GB): 127 tok/s Mac /
|
| 45 |
19–21 iPhone. Smaller; keep if you want the head at full precision.
|
| 46 |
|
|
|
|
| 84 |
|
| 85 |
Conversion script (self-contained) + method page in the zoo:
|
| 86 |
[`conversion/export_qwen3_5_decode_pipelined.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_qwen3_5_decode_pipelined.py)
|
| 87 |
+
(`int8hu --head-sym --hf-id Qwen/Qwen3.5-2B`) ·
|
| 88 |
[`knowledge/pipelined-engine.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/pipelined-engine.md)
|