docs: descriptive model card
Browse files
README.md
CHANGED
|
@@ -63,11 +63,13 @@ entrypoint, `prefill`, with a static query width of 64
|
|
| 63 |
| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
|
| 64 |
| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
|
| 65 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED β see below** |
|
|
|
|
| 66 |
|
| 67 |
Each folder holds `<name>.aimodel/` (`main.mlirb` β 3.96 GB, `main.hash`, asset
|
| 68 |
`metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
|
| 69 |
32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
|
| 70 |
-
18,569 B).
|
|
|
|
| 71 |
|
| 72 |
**The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
|
| 73 |
exported graph β identical function signatures, identical state descriptors, identical
|
|
@@ -191,8 +193,23 @@ does not widen the sliding layers' 512-token window.
|
|
| 191 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED β measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
|
| 192 |
| `ctx8192/β¦_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β same weights, same graph, wider manifest integer. No run above 4096 tokens. |
|
| 193 |
| `ctx16384/β¦_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β as above. |
|
|
|
|
| 194 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED β does not load on any current code path.** |
|
| 195 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 196 |
### Why the non-`tbl` decode bundle is blocked
|
| 197 |
|
| 198 |
It declares a **different third input** from the `_tbl` family:
|
|
|
|
| 63 |
| `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
|
| 64 |
| `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
|
| 65 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED β see below** |
|
| 66 |
+
| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | 3,990,690,638 | 16384 | main + prefill | **EXPERIMENTAL β see below** |
|
| 67 |
|
| 68 |
Each folder holds `<name>.aimodel/` (`main.mlirb` β 3.96 GB, `main.hash`, asset
|
| 69 |
`metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
|
| 70 |
32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
|
| 71 |
+
18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl`
|
| 72 |
+
(35,045 B).
|
| 73 |
|
| 74 |
**The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
|
| 75 |
exported graph β identical function signatures, identical state descriptors, identical
|
|
|
|
| 193 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED β measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
|
| 194 |
| `ctx8192/β¦_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β same weights, same graph, wider manifest integer. No run above 4096 tokens. |
|
| 195 |
| `ctx16384/β¦_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β as above. |
|
| 196 |
+
| `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL β built, unmeasured.** Gate: a Mac-side oracle/parity check plus a device benchmark, neither of which has been run. Do not route production traffic to it. |
|
| 197 |
| `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED β does not load on any current code path.** |
|
| 198 |
|
| 199 |
+
### About the `w4a8` bundle
|
| 200 |
+
|
| 201 |
+
Same QAT int4 per-block-32 weights and the same graph as
|
| 202 |
+
`gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64`; the difference is an int8
|
| 203 |
+
quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic
|
| 204 |
+
social-media-style samples (the corpus ships in the folder). It was built as a prefill/TTFT
|
| 205 |
+
lever, and its manifest context is 16384.
|
| 206 |
+
|
| 207 |
+
The caveat is load-bearing and comes from the compression toolchain's own documentation: the
|
| 208 |
+
runtime fast path is `W_INT8 Γ A_INT8`, and a float weight path executes in floating point
|
| 209 |
+
*regardless* of activation quantization. These weights use int4 scale-multiply dequant, not
|
| 210 |
+
an int8 LUT, so the expected TTFT gain may be **zero or negative**. This bundle exists to be
|
| 211 |
+
measured, not because a win is predicted.
|
| 212 |
+
|
| 213 |
### Why the non-`tbl` decode bundle is blocked
|
| 214 |
|
| 215 |
It declares a **different third input** from the `_tbl` family:
|