README: stable bundles
Browse files
README.md
CHANGED
|
@@ -256,3 +256,39 @@ model and **the Gemma terms apply downstream**, which is why this repo declares
|
|
| 256 |
[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
|
| 257 |
travel with any redistribution of these bundles, including the gather-table sidecar, which
|
| 258 |
is derived from the same weights. The contribution here is the conversion, not the weights.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 256 |
[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations
|
| 257 |
travel with any redistribution of these bundles, including the gather-table sidecar, which
|
| 258 |
is derived from the same weights. The contribution here is the conversion, not the weights.
|
| 259 |
+
|
| 260 |
+
|
| 261 |
+
## `stable/` — shape-stable decode contract
|
| 262 |
+
|
| 263 |
+
> ⚠️ **EXPERIMENTAL — built, not yet measured on a Mac.** A re-export of the same weights
|
| 264 |
+
> and the same quantization onto a decode/prefill contract in which no input shape moves
|
| 265 |
+
> between steps, which removes the per-generated-token memory growth described in the
|
| 266 |
+
> known issue above. It requires a host that feeds `position_ids` as the **absolute
|
| 267 |
+
> positions of the S new tokens only** (host support in progress); a host that feeds the
|
| 268 |
+
> growing `0..N` prefix will write the KV cache at the wrong offset. No throughput or
|
| 269 |
+
> footprint numbers have been taken yet.
|
| 270 |
+
|
| 271 |
+
| Bundle | Bytes | Context | Functions | Producer | Created |
|
| 272 |
+
|---|---:|---|---|---|---|
|
| 273 |
+
| `stable/gpu-pipelined/gemma4_e4b_qat_decode_int4lin_tbl_pf64_stable_c16384` | 3,989,939,812 | 16384 | main+prefill | coreai-core 1.0.0b2 | 20260818T004019Z |
|
| 274 |
+
|
| 275 |
+
**Contract.** Both entrypoints take four inputs and two states, all statically shaped:
|
| 276 |
+
|
| 277 |
+
```
|
| 278 |
+
main IN input_ids Int32 1x1 | position_ids Int32 1x1 | ple_table Int8 V x (L*ld) | ple_scale Float32 V
|
| 279 |
+
prefill IN input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics)
|
| 280 |
+
ST keyCache / valueCache Float16 [slots, 1, n_kv, 16384, 512]
|
| 281 |
+
OUT logits Float16 1 x S x 262144
|
| 282 |
+
```
|
| 283 |
+
|
| 284 |
+
`position_ids` carries the absolute position of each of the S tokens in the call, and
|
| 285 |
+
`position_ids[0,0]` is also the cache slot the K/V for those tokens is written at. The
|
| 286 |
+
attention mask (causality, unwritten slots, and the 512-token sliding window) is derived
|
| 287 |
+
inside the graph from those positions, so there is no mask input to bind. The KV state's
|
| 288 |
+
sequence extent is a literal 16,384 rather than a dynamic dimension, so a host that
|
| 289 |
+
resolves its cache strategy from the state descriptor will allocate the full cache up
|
| 290 |
+
front (0.50 GB for E2B, 1.61 GB for E4B) instead of growing it — that fixed cost is the
|
| 291 |
+
trade for the unbounded growth it replaces.
|
| 292 |
+
|
| 293 |
+
Unlike the `ctx*/` folders, the context ceiling here is **encoded in the graph**: a
|
| 294 |
+
different window needs a different export, not a manifest edit.
|