Gemma 4 q4_0 QAT Text Decoder — Karume

This is an unofficial conversion of google/gemma-4-E2B-it, google/gemma-4-E2B-it-qat-q4_0-unquantized, google/gemma-4-E4B-it, google/gemma-4-E4B-it-qat-q4_0-unquantized, google/gemma-4-12B-it, google/gemma-4-12B-it-qat-q4_0-unquantized, google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant, google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant and google/gemma-4-12B-it-qat-q4_0-unquantized-assistant for Karume. It is not affiliated with or endorsed by Google.

What is this

A chat distribution: the text decoder of google/gemma-4-E2B-it-qat-q4_0-unquantized / google/gemma-4-E4B-it-qat-q4_0-unquantized / google/gemma-4-12B-it-qat-q4_0-unquantized, converted into the WebGPU inference runtime Karume's container format (a .krm part sequence whose first part carries the graph and model descriptors). Runs as-is in the browser and in Deno — string in, string out.

Tokenizer and chat format ship with the weights. chat() renders the turns, encodes them, samples, and decodes incrementally, so callers hand over messages and read back text fragments as they are decided.
Every weight keeps the checkpoint's own 4-bit grid. The linear weights, the embedding table (also the output head) and the per-layer embeddings of e2b / e4b are stored as the q4_0 codes with one step per group of 32 values (packed int4) — not rounded again.
Speculative decoding: e2b / e4b / 12b also ship Google's QAT multi-token-prediction drafter, which borrows the main model's key/value states and embedding table; pass speculative: {} when loading to use it.
Per-layer embeddings (e2b / e4b) are container assets, not graph weights. They are gathered on the host and handed to the graph as an input, which keeps them out of the GPU's resident set; the loader reads only the vocabulary ranges a conversation actually touches. 12b has no per-layer embeddings.
Context: the default capacity is 4096 tokens per conversation, and the loader accepts any capacity between the chunk length in use and the model's declared position limit (131072 for e2b / e4b, 262144 for 12b); rotary cos/sin are generated on the host per chunk from the declared rope parameters, so no position table ships. Prefill runs in chunks of 768 rows by default, and the chunk length itself can be raised up to 768 — the traced upper bound of the chunk symbol, which the graph does not carry.
Not multimodal here. The published checkpoints also carry multimodal front ends (vision and audio towers for e2b / e4b; encoder-free vision and audio embedders for 12b); this distribution contains none of them, and cannot take images or audio.
Not readable by transformers (it's a different container with an embedded graph); the reader is a pipeline that implements gemma4/1.
Exporter used for the conversion: karume/0.16.0. The distribution manifest is karume.json (karume/5).

Base weights and attribution

Converted from Google's quantization-aware-trained (QAT, Q4_0) checkpoints — the original checkpoints are not distributed here.

e2b: the weights are google/gemma-4-E2B-it-qat-q4_0-unquantized, Google's QAT (Q4_0) checkpoint of the base model google/gemma-4-E2B-it, which is also the source of the tokenizer, the chat template and the generation defaults; both are licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
e4b: the weights are google/gemma-4-E4B-it-qat-q4_0-unquantized, Google's QAT (Q4_0) checkpoint of the base model google/gemma-4-E4B-it, which is also the source of the tokenizer, the chat template and the generation defaults; both are licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
12b: the weights are google/gemma-4-12B-it-qat-q4_0-unquantized, Google's QAT (Q4_0) checkpoint of the base model google/gemma-4-12B-it, which is also the source of the tokenizer, the chat template and the generation defaults; both are licensed Apache 2.0 (license / full text; a verbatim copy is in LICENSE.md).
e2b drafter: google/gemma-4-E2B-it-qat-q4_0-unquantized-assistant, the QAT multi-token-prediction drafter head, licensed Apache 2.0 under the same terms. It reads the key/value states and the embedding table of the main model rather than carrying its own, and runs its output head over the full vocabulary (a clustered sparse head an assistant checkpoint carries is not used).
e4b drafter: google/gemma-4-E4B-it-qat-q4_0-unquantized-assistant, the QAT multi-token-prediction drafter head, licensed Apache 2.0 under the same terms. It reads the key/value states and the embedding table of the main model rather than carrying its own, and runs its output head over the full vocabulary (a clustered sparse head an assistant checkpoint carries is not used).
12b drafter: google/gemma-4-12B-it-qat-q4_0-unquantized-assistant, the QAT multi-token-prediction drafter head, licensed Apache 2.0 under the same terms. It reads the key/value states and the embedding table of the main model rather than carrying its own, and runs its output head over the full vocabulary (a clustered sparse head an assistant checkpoint carries is not used).
Changes made here (also listed in NOTICE.md, per Apache 2.0 §4(b)): the text decoder was extracted and re-expressed in the Karume container format; the linear weights and the embedding table keep the checkpoint's own 4-bit grid, stored as packed int4 (group 32: its codes and one step per group, refitted from the stored values) rather than rounded again; the per-layer embeddings of e2b / e4b were moved out of the graph into container assets that the host gathers, on the same grid; the exit was narrowed to the selected rows' logits and hidden states; rotary cos/sin arrive as host-generated inputs; the drafters' weights keep their own grid the same way. No retraining and no fine-tuning.
What the model can and cannot do — capabilities, limitations, intended use, evaluations and safety guidance — is documented on the upstream model card, and this repository does not restate it: read google/gemma-4-E2B-it before using it.

Models

Model	Pipeline	Quants	Default quant
`e2b` (default)	`gemma4/1`	`i4` / `i4-gemvpar` / `i4-fast`	`i4`
`e4b`	`gemma4/1`	`i4` / `i4-gemvpar` / `i4-fast`	`i4`
`12b`	`gemma4/1`	`i4`	`i4`

model selects one of these; omitted, it is e2b. quant defaults to that model's own default quant.

Usage

import { Pipeline } from "jsr:@karume/models/gemma";

await using pipeline = await Pipeline.fromPretrained({
  repo: "hdae/karume-gemma4-qat",
  // Pin a commit for reproducible builds — without it you track `main`, and a future
  // repo update (renamed files, new manifest format) may break your app.
  // Copy the full hash from this repo's "Files and versions" tab:
  // revision: "<full commit sha>",
}, {
  // model: "e2b", // default — available: 12b / e2b / e4b
  // quant: "i4", // default — available: i4 / i4-fast / i4-gemvpar
  // speculative: {}, // load the MTP drafter — available for: e2b / e4b / 12b
});

const stream = pipeline.chat([
  { role: "user", content: "What is the capital of France?" },
], {
  maxNewTokens: 128,
  // sampler: { temperature: 0 }, // greedy — overrides the recommended default below
});

// Fragments arrive as the decoder settles them (multi-byte characters are held back
// until they are complete).
for await (const text of stream) Deno.stdout.write(new TextEncoder().encode(text));
console.log(await stream.done); // { reason: "eos" | "max-tokens" | "capacity" | … }

Messages are plain system / developer / user / assistant turns with text content; content parts such as images or audio are rejected rather than silently dropped. For function calling and thinking, use new ChatSession(pipeline, { tools, enableThinking }): a turn can end on a tool call, and sendToolResults continues it. With no sampler in the request, generation uses this repository's recommended default: temperature 1.0, top-k 64, top-p 0.95. Weights are fetched once and cached (verified against karume.json's size / sha256).

Long context

chat() takes a longer context through capacity, for example { capacity: 32768, maxNewTokens: 256 }. Measured on an RTX 3080 Ti under Node + Dawn with greedy decoding, except where the text says Chrome.

Model	Quant	Compared with upstream up to	Facts found in a long document
`e2b`	`i4-fast`	128K prompt tokens	158 of 160 up to 32K, 25 of 32 at 64K, 23 of 32 at 128K
`12b`	`i4`	32K prompt tokens	160 of 160 up to 32K

The conversion follows upstream. Against transformers running the same QAT checkpoint in f32, KL on the logits does not grow with position (at most 7.0e-4 on e2b, 8.7e-3 on 12b), the top-1 token agrees on every compared prefill row, and all 5 prompts run through both gave the same text.
e2b itself loses facts in long documents. The test places 8 short facts in a document and asks for them. From 64K, e2b misses some, mostly on varied prose. Upstream gave the same wrong answers on the 3 such prompts compared, so this is the model, not the conversion. For answers that depend on the whole document, treat 32K as the practical limit of e2b.
Not covered: tasks other than finding short facts, such as summaries or reasoning over a long text; non-greedy sampling; 12b above 32K. e4b was not checked.

Decode, ms per token	32K prompt	128K prompt
`e2b` `i4`	110	385
`e2b` `i4-fast`	12	26
`e2b` `i4`, Chrome on an RTX 5070 Ti	147	444
`12b` `i4`	183	not run

i4-fast is the quant to pick for long context: decode time grows far more slowly than on i4, and it gave the same tokens as i4 in these runs. It was not run at long context in Chrome. The RTX 3080 Ti rows were under thermal slowdown.

Prefill is the wait before the first token: in Chrome on the RTX 5070 Ti, e2b i4 took 22 s for a 32K prompt and 175 s for a 128K prompt. 12b i4 took about 110 s at 32K on the RTX 3080 Ti. Whole-GPU memory held about 3,000 MiB on e2b i4 at capacity 32768, about 4,900 MiB at capacity 131072, and about 10,500 MiB on 12b i4 at capacity 32768; short peaks about 550 MiB above that were sampled at capacity 65536.

Model: e2b

Quants

Quant	What it is	Download	Weights	Compute
`i4` (default)	QAT int4 grid, drafter included — The q4_0 QAT checkpoints' own 4-bit grid (packed int4, group 32) for every linear weight, embedding table and drafter weight; GEMV and attention use the reference summation order.	2.75 GiB (1.38 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	—
`i4-gemvpar`	QAT int4 grid with parallel GEMV and attention — Same weights as i4; parallel GEMV (not the int4 output head) and attention. Rounding and generated tokens can differ. Select i4 for the reference order. Faster than i4 on the tested E2B device.	2.75 GiB (1.38 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	`linearGemvReduce` = `parallel` / `stateAttentionReduce` = `parallel`
`i4-fast`	QAT int4 grid with parallel GEMV, RMS fusion and fused attention — Same weights as i4; parallel GEMV (not the int4 output head), RMS-add fusion, fused attention. Use i4 for reference summation. Requires fusion-option support. Faster than i4 on the tested E2B device.	2.75 GiB (1.38 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	`linearGemvReduce` = `parallel` / `fuseRmsNormAdd` = `true` / `stateAttentionReduce` = `parallel-fused`

If no quant is given, it runs as i4 (this model's recommended default). Per-file size and sha256 live in karume.json — verify against that at the fetch layer. Dtype labels use the runtime's storage dtype vocabulary (f16 / bf16 / i8 / i4 / i2), not the fp16 spelling common elsewhere in the ecosystem. Weights ship as Karume container files (.krm), split into numbered parts; a part is fetched and verified on its own.

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
stop tokens come from the tokenizer asset in this repository, not from the caller
device limits: maxStorageBufferBindingSize ≥ 268,435,456 B (the largest single tensor must bind)

Model: e4b

Quants

Quant	What it is	Download	Weights	Compute
`i4` (default)	QAT int4 grid, drafter included — The q4_0 QAT checkpoints' own 4-bit grid (packed int4, group 32) for every linear weight, embedding table and drafter weight; GEMV and attention use the reference summation order.	4.40 GiB (1.65 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	—
`i4-gemvpar`	QAT int4 grid with parallel GEMV and attention — Same weights as i4; parallel GEMV (not the int4 output head) and attention. Rounding and generated tokens can differ. Select i4 for the reference order. Faster than i4 on the tested E4B device.	4.40 GiB (1.65 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	`linearGemvReduce` = `parallel` / `stateAttentionReduce` = `parallel`
`i4-fast`	QAT int4 grid with parallel GEMV, RMS fusion and fused attention — Same weights as i4; parallel GEMV (not the int4 output head), RMS-add fusion, fused attention. Use i4 for reference summation. Requires fusion-option support. Faster than i4 on the tested E4B device.	4.40 GiB (1.65 GiB of assets, read on the host)	`model` = `i4` / `drafter` = `i4`	`linearGemvReduce` = `parallel` / `fuseRmsNormAdd` = `true` / `stateAttentionReduce` = `parallel-fused`

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 131072 can be chosen at load time
prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
stop tokens come from the tokenizer asset in this repository, not from the caller
device limits: maxBufferSize ≥ 536,870,912 B / maxStorageBufferBindingSize ≥ 536,870,912 B (the largest single tensor must bind)

Model: 12b

Quants

Quant	What it is	Download	Weights	Compute
`i4` (default)	QAT int4 grid, drafter included — The q4_0 QAT checkpoints' own 4-bit grid (packed int4, group 32) for every linear weight, embedding table and drafter weight; GEMV and attention use the reference summation order.	7.19 GiB	`model` = `i4` / `drafter` = `i4`	—

Generation

Derived from the exported graph and the checkpoint's own generation_config.json, and checked against each other when this repository was assembled.

context: 4096 tokens per conversation (prompt + generated) by default; any capacity from the chunk length in use up to 262144 can be chosen at load time
prefill chunk: 768 rows per step by default, up to 768 (the traced upper bound of the chunk symbol)
recommended sampler: temperature 1.0, top-k 64, top-p 0.95 — used when a request omits sampler; pass { temperature: 0 } for greedy decoding
stop tokens come from the tokenizer asset in this repository, not from the caller
device limits: maxBufferSize ≥ 536,870,912 B / maxStorageBufferBindingSize ≥ 536,870,912 B (the largest single tensor must bind)

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for hdae/karume-gemma4-qat

Base model

google/gemma-4-12B

Finetuned

google/gemma-4-12B-it

Quantized

(395)

this model