HiramKHackenbacker commited on
Commit
1998b7e
Β·
verified Β·
1 Parent(s): 451139d

docs: descriptive model card

Browse files
Files changed (1) hide show
  1. README.md +18 -1
README.md CHANGED
@@ -63,11 +63,13 @@ entrypoint, `prefill`, with a static query width of 64
63
  | `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
64
  | `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
65
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED β€” see below** |
 
66
 
67
  Each folder holds `<name>.aimodel/` (`main.mlirb` β‰ˆ 3.96 GB, `main.hash`, asset
68
  `metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
69
  32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
70
- 18,569 B).
 
71
 
72
  **The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
73
  exported graph β€” identical function signatures, identical state descriptors, identical
@@ -191,8 +193,23 @@ does not widen the sliding layers' 512-token window.
191
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED β€” measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
192
  | `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β€” same weights, same graph, wider manifest integer. No run above 4096 tokens. |
193
  | `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β€” as above. |
 
194
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED β€” does not load on any current code path.** |
195
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
196
  ### Why the non-`tbl` decode bundle is blocked
197
 
198
  It declares a **different third input** from the `_tbl` family:
 
63
  | `ctx8192/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,830 | 8192 | main + prefill | same weights, wider manifest |
64
  | `ctx16384/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` | 3,989,986,844 | 16384 | main + prefill | same weights, wider manifest |
65
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | 3,989,205,604 | 4096 | main | **BLOCKED β€” see below** |
66
+ | `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | 3,990,690,638 | 16384 | main + prefill | **EXPERIMENTAL β€” see below** |
67
 
68
  Each folder holds `<name>.aimodel/` (`main.mlirb` β‰ˆ 3.96 GB, `main.hash`, asset
69
  `metadata.json`), a bundle-level `metadata.json`, and `tokenizer/` (`tokenizer.json`
70
  32,169,626 B, `tokenizer_config.json`, `generation_config.json`, `chat_template.jinja`
71
+ 18,569 B). The `w4a8` folder additionally ships its `calibration_corpus.jsonl`
72
+ (35,045 B).
73
 
74
  **The three `_tbl_pf64` folders are the same weights.** `--max-ctx` does not change the
75
  exported graph β€” identical function signatures, identical state descriptors, identical
 
193
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64` (ctx 4096) | **QUALIFIED β€” measured, viable, not a default.** 10/10 parse, 10/10 enum-clean on the harness above. Slow and memory-bound at the 16 GB tier; it earns a slot only where Gemma's output specifically is wanted. |
194
  | `ctx8192/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β€” same weights, same graph, wider manifest integer. No run above 4096 tokens. |
195
  | `ctx16384/…_tbl_pf64` | **QUALIFIED BY EQUIVALENCE, UNMEASURED AT DEPTH** β€” as above. |
196
+ | `w4a8/gemma4_e4b_qat_decode_int4lin_a8_tbl_pf64` | **EXPERIMENTAL β€” built, unmeasured.** Gate: a Mac-side oracle/parity check plus a device benchmark, neither of which has been run. Do not route production traffic to it. |
197
  | `gpu-pipelined/gemma4_e4b_qat_decode_int4lin` | **BLOCKED β€” does not load on any current code path.** |
198
 
199
+ ### About the `w4a8` bundle
200
+
201
+ Same QAT int4 per-block-32 weights and the same graph as
202
+ `gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64`; the difference is an int8
203
+ quantize/dequantize pair on the inputs of every `F.linear`, calibrated on 128 synthetic
204
+ social-media-style samples (the corpus ships in the folder). It was built as a prefill/TTFT
205
+ lever, and its manifest context is 16384.
206
+
207
+ The caveat is load-bearing and comes from the compression toolchain's own documentation: the
208
+ runtime fast path is `W_INT8 Γ— A_INT8`, and a float weight path executes in floating point
209
+ *regardless* of activation quantization. These weights use int4 scale-multiply dequant, not
210
+ an int8 LUT, so the expected TTFT gain may be **zero or negative**. This bundle exists to be
211
+ measured, not because a win is predicted.
212
+
213
  ### Why the non-`tbl` decode bundle is blocked
214
 
215
  It declares a **different third input** from the `_tbl` family: