gemma4-31b-it-dflash
google/gemma-4-31B-it on a QuietBox 2 (P300x2, 4 Blackhole chips) with vLLM. The default profile serves one user on a resident decode engine (one persistent program per chip per decode step) with DFlash2 speculative decoding, up to 262K context. batch serves up to 32 users on a resident batch engine that keeps per-user decode fast at long context, up to 262K. Both profiles use a paged-bucket prefill: a fixed set of prefill programs built at start-up, so device memory no longer grows with the number of distinct prompt lengths served.
Runs on p300x2 — see the serve profiles below.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
Quickstart
tt-model pull changh95/gemma4-31b-it-dflash --with-weights
tt-model serve changh95/gemma4-31b-it-dflash
pull --with-weights downloads the Docker image and the google/gemma-4-31B-it weights at 842da3794eaa0b77d5f08bae87a17459d91ff475 (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.
At a glance
- Weights: google/gemma-4-31B-it @
842da37; drafter RedHatAI/gemma-4-31B-it-speculator.dflash2 @b0cd01b - Hardware: P300x2 (QuietBox 2), one (1, 4) Blackhole mesh, tensor-parallel over 4 chips
- Profiles:
single-user(default),batch - Context: 262,144 tokens on every profile
- Weights on device: BF4 (bfloat4_b) projections, BF8 LM head and KV cache, as validated
- Thinking is on by default. Turn it off per request with
"chat_template_kwargs": {"enable_thinking": false}. - Default sampling: the model's generation config (temperature 1.0, top_p 0.95) with top_k 20. Pass
"temperature": 0for greedy. single-userspeculates on greedy and on sampled requests (temperature, top_p, top_k). Penalties and top_k above 32 are served without speculation. Speculative output can differ from plain decoding at near-ties.single-userrefuses (HTTP 400) what its speculative path cannot apply: logprobs, structured outputs (response_format, guided decoding),bad_words,allowed_token_ids,min_tokens,logit_bias,min_p.batchserves logprobs (on the host-sampled step; see Known limitations).- Tool calling:
single-userparses tool calls (vLLM'sgemma4parser,tool_choice: "auto").batchhas no tool calling. batch(update 3): the resident batch engine. It picks its decode width by the number of live users (R = 1 / 8 /- and does not speculate.
Update 2: paged-bucket prefill
- Both profiles now build their prefill programs for a fixed set of length buckets at start-up. Update 1 compiled a new program set for every distinct prompt length and kept it for the life of the server, about 0.24 MB of device memory per chip per new length. Update 2's program set is bounded by the fixed buckets.
- Decoding is unchanged. In the gating runs, update 2's outputs matched update 1's on the same prompts.
- Start-up cost, measured with the kernels already compiled: about 5 s more for
single-user, which also pre-builds the drafter's seed programs (about 6.5 MB per chip). The extra compile time on a first start has not been measured yet. - A prompt longer than one 8,192-token chunk builds a few programs the first time a new continuation length class is
seen (lazy, on every profile).
single-userignoresGEMMA4_PREFILL_WARM_CONT=1(its continuation programs are always built on first use, and its drafter seed pre-warm is always on);batchrefuses to start with it set. - Measured first-sight cost of a new continuation length class, on a fresh kernel cache: about +1.05 s TTFT on
single-user(kernel compilation, once per host per class); about 10 ms once the kernels are on disk. Not measured onbatch.
Update 3: the resident batch engine (batch)
--profile batch now selects the resident batch engine. The previous batch-b32 profile was removed; the update-2 revision keeps it.
- Start-up, measured in update 3.1's proof (in the package container, the same code): 120 s to ready on a warm
start. The first
batchstart on a host that had already runsingle-useronce took 241 s, including the one-time conversion ofbatch's own copy of the engine weights (converted in 55 s; 22.8 GB more disk). A fresh host that startsbatchfirst also compiles the shared kernels: that start is not measured.single-user's weight copy is untouched. batchserves logprobs, repetition / presence / frequency penalties and logit_bias on its host-sampled step, each alone and alongside 31 greedy users (checked in this release's proof); see Known limitations.- One long request among short ones: 32 users with 1K-token prompts (512 output tokens) had a TPOT of 35.6 ms; with one of them replaced by a 259,000-token request, its prefill already done, the 31 short users' TPOT was 42.5 ms (2026-11-15, the same engine before the paged-bucket prefill was added; chip clocks were below 1,323 MHz for 4–19% of the 32-user windows).
- Reliability: 0 hangs in 491,520 long-context launches (≤ 6.1e-6 per launch, 95%), in a 6.5 h soak of
batch's engine at 262K context.
Real prompts. vllm bench serve (tt-inference-server's client) with the real-prompt builder's chat prompts: a
coding, writing or math task after a real context, cut so the chat template renders it to exactly the stated length;
thinking on (the server default), greedy, 256 output tokens, warm-ups = concurrency. Times are as measured; points
where chip clocks dropped during decode carry a footnote with the per-chip share of decode time below 1,323 MHz.
| ISL | Concurrency | TPOT (ms) | TTFT (ms) |
|---|---|---|---|
| 128 | 1 | 10.2 | 64 |
| 128 | 8 | 14.3 | 392 |
| 128 | 32 | 26.8 | 1,465 |
| 1,024 | 1 | 11.4 | 145 |
| 1,024 | 8 | 16.7[1] | 1,054 |
| 1,024 | 32 | 38.0[2] | 2,905 |
| 8,192 | 1 | 11.7[3] | 1,181 |
| 32,768 | 1 | 12.4[4] | 5,635 |
- [1] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 10.48 / 10.48 / 0.04 / 10.48. Times as measured, not corrected.
- [2] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 14.17 / 17.48 / 0 / 17.48. Times as measured, not corrected.
- [3] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 0 / 17.64 / 26.46 / 17.64. Times as measured, not corrected.
- [4] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 14.95 / 34.24 / 28.94 / 24.11. Times as measured, not corrected.
- The clock footnotes cover the decode windows (TPOT) only.
Measured 2026-11-16 09:28–09:31 (KST), outside the package container on the same files (byte-identical engine and adapter code); this release's proof measures the packaged image at the same points.
Which profile
- One user:
single-user. - 2 to 32 concurrent users, or several users at long context:
batch.
Get the drafter (single-user)
hf download RedHatAI/gemma-4-31B-it-speculator.dflash2 --revision b0cd01b4918aae6b6388272950c4f9851734674b
Serve
tt-model serve changh95/gemma4-31b-it-dflash # default profile: single-user
tt-model serve changh95/gemma4-31b-it-dflash --profile batch
tt-model profiles changh95/gemma4-31b-it-dflash
- First start on a host (
single-user): about 12.5 min to ready. It converts the resident engine's weights once (about 70 s: 22 GB from the 59 GB checkpoint) and compiles kernels once (about 9.5 min). - Later starts: about 2.5 min, including a sha256 check of the converted weights (seconds when they are in the page cache).
- These start times were measured on update 1. Update 2 adds its prefill warm-up (see above);
batch's start-up is in the update 3 section. - Disk: about 31 GB under
~/.cache/tt-model/gemma4-31b-it-dflash(22 GB converted weights, 9 GB compiled kernels), plus 22 GB forbatch's own weight copy once it has started. A package update that changes an engine converts that engine's copy again;single-user's copy is unchanged by update 3.
How to query
curl http://localhost:20000/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "google/gemma-4-31B-it",
"messages": [{"role": "user", "content": "Write a haiku about Blackhole chips."}],
"max_tokens": 256, "temperature": 0,
"chat_template_kwargs": {"enable_thinking": false}
}'
Known limitations
- Code-generation quality (BF4 weights). On the same evaluation harness, run through update 1's
batchdecode path, a build of this model with BF8 weights scores higher on code generation:- MBPP: 0.566 (this build) vs 0.658, a significant gap (paired McNemar test, Holm-adjusted p ≈ 1e-6).
- HumanEval: 0.835 vs 0.884, not significant.
- About half of the MBPP gap is answers that never close their code block, mostly because they run long or repeat until the 512-token limit.
- Long thinking traces repeat more often: 12.8% vs 8.8% are degenerate.
- GSM8K, IFEval, MMLU-Pro, MATH-500 and RULER (8K) are within noise.
- Host-sampled requests in
batch. A request that needs CPU sampling (repetition, presence or frequency penalties, logprobs, or top_k above the device limit of 32) moves sampling for every concurrent request to the CPU while it is active.- At 32 users a decode step takes 277.5 ms while one of them carries a repetition penalty (the other 31
default-sampled), against 26.9 ms when all 32 are greedy (update 3.1's proof, same code, with
OMP_NUM_THREADS=8, which this package sets). - Greedy requests and the default device sampling are unaffected.
- The full fix belongs in the vLLM TT plugin.
- At 32 users a decode step takes 277.5 ms while one of them carries a repetition penalty (the other 31
default-sampled), against 26.9 ms when all 32 are greedy (update 3.1's proof, same code, with
batch: greedy text depends on load. The engine changes its decode width (R = 1 / 8 / 32) as users join and leave, and the same greedy prompt can produce different text depending on how many other requests are running. At a fixed width, greedy text was bitwise stable under co-tenant churn for prompts that prefill in one step. A prompt that is chunked alongside others can see small differences.batch: prefix caching is off (as on every profile): repeated prefixes are recomputed.- Device memory. Lowest free DRAM per chip measured on
batch: 2.85 GB, briefly, right after a burst of 1K-prompt prefills at up to 32 users with a 200K-token request live (a 6.5 h, 491K-launch soak). At steady state about 6.3 GB, flat over the soak. - Thermal clock dips. On a QuietBox 2 chip 1 can lower its clock after about 20 s of sustained fast decode, raising per-token time by up to about 1.3–1.45× while it lasts (measured on earlier builds). Every table on this card notes the clock dips during its run.
batch: host decode guard. The engine refuses a decode mix whose conservative attention bound exceeds 40 units per core. With its KV pool (4,320 blocks, about 553K positions) no reachable mix comes near it (at most 25 of 40); it would matter only with a pool about 4.5 times larger (above 19,648 blocks).- Update 2 evidence scope. The paged-bucket prefill's runtime guard and its gating runs check program-cache growth and outputs; they did not record device buffer addresses. The drafter seed pre-warm's first-start (cold kernel cache) time is not measured.
Results (tt-inference-server --workflow benchmarks)
The first three tables were measured in one session on 2026-11-17 on update 3.1's image (7b6dd89c97bd), whose code is identical to this release's (update 3.2 changes only the manifest: this card and the profile list): each profile in its own tt-model serve --profile … container on port 8000, with the same tt-inference-server invocations, specs and points as update 1's tables of 2026-11-11, which remain in the update-2 revision on the Hub. The fourth was measured in a second session on 2026-11-17, on the same image.
single-user
| ISL | OSL | Concurrency | Num Requests | TTFT (ms) | TPOT (ms) | Tput User (t/s/u = 1000/TPOT) | Tput Output (t/s) |
|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 8 | 58.4 | 4.5 | 224.6 | 205.0 |
| 128 | 1024 | 1 | 4 | 59.1 | 3.8 | 266.0 | 262.2 |
| 1024 | 128 | 1 | 4 | 137.7 | 8.7 | 115.1 | 103.1 |
| 2048 | 128 | 1 | 4 | 281.0 | 8.9 | 112.1 | 90.5 |
| 4096 | 128 | 1 | 4 | 565.8 | 8.8 | 113.0 | 75.7 |
| 8192 | 128 | 1 | 2 | 1168.9 | 7.5 | 132.9 | 60.2 |
| 8192 | 1024 | 1 | 2 | 1171.5 | 8.7 | 114.6 | 101.4 |
| 10000 | 1024 | 1 | 2 | 1483.6 | 3.9 | 258.5 | 188.2 |
| 16384 | 128 | 1 | 2 | 2500.8 | 11.9 | 83.9 | 31.9 |
| 32768 | 128 | 1 | 1 | 5617.2 | 8.5 | 117.5 | 19.1 |
| 65536 | 128 | 1 | 1 | 13642.1 | 4.9 | 205.2 | 9.0 |
| 131072 | 128 | 1 | 1 | 36924.4 | 5.0 | 201.8 | 3.4 |
| 196608 | 128 | 1 | 1 | 69572.7 | 5.1 | 195.8 | 1.8 |
| 262016 | 128 | 1 | 1 | 111630.0 | 5.7 | 175.0 | 1.1 |
Run 2026-11-17 15:42–15:50 (KST), tt-inference-server 0.26.1 --workflow benchmarks against tt-model serve --profile single-user; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 7 of 14 points (chips 0, 1, 2, 3; up to 36% of decode time: 1024/128 c1, 2048/128 c1, 4096/128 c1, 8192/128 c1, 8192/1024 c1, 10000/1024 c1, 16384/128 c1).
Greedy continuations of random prompts repeat and draft unusually well, so these rates are above what real text gets;
the realistic-prompt numbers below are the better guide.
single-user, realistic prompts (tt-inference-server --spec-decode)
| Prompts (SPEED-Bench) | Concurrency | Num Requests | TTFT (ms) | TPOT (ms) | Tput User (t/s/u = 1000/TPOT) | Tput Output (t/s) | Tokens per verify (of 8) |
|---|---|---|---|---|---|---|---|
| coding | 1 | 80 | 121.6 | 4.9 | 204.3 | 190.2 | 4.60 |
| throughput_32k | 1 | 32 | 6641.3 | 11.1 | 89.8 | 69.5 | 2.64 |
Run 2026-11-17 15:58–16:28 (KST), tt-inference-server 0.26.1 --workflow benchmarks --spec-decode against tt-model serve --profile single-user: SPEED-Bench prompts (AIPerf; the requests carry no sampling fields, so the server's defaults apply: temperature 1.0, top_p 0.95, top_k 20, thinking on). Each answer runs to its natural end (throughput sets capped at 8,192 tokens). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz for coding 0.1% of decode time; throughput_32k 0.5% of decode time (76% of prefill).
batch (the resident batch engine)
| ISL | OSL | Concurrency | Num Requests | TTFT (ms) | TPOT (ms) | Tput User (t/s/u = 1000/TPOT) | Decode t/s/u (1000/median ITL)¹ | Tput Output (t/s) |
|---|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 8 | 65.1 | 10.1 | 99.3 | 98.9 | 95.2 |
| 128 | 128 | 32 | 256 | 1471.8 | 26.9 | 37.2 | 37.2 | 838.6 |
| 128 | 1024 | 1 | 4 | 64.9 | 10.8 | 92.5 | 89.6 | 92.1 |
| 128 | 1024 | 32 | 128 | 1464.0 | 29.3 | 34.1 | 34.5 | 1041.8 |
| 1024 | 128 | 1 | 4 | 148.1 | 11.4 | 88.0 | 88.0 | 80.4 |
| 1024 | 128 | 32 | 128 | 3026.1 | 41.8 | 23.9 | 30.6 | 491.4 |
| 2048 | 128 | 1 | 4 | 291.6 | 11.4 | 87.8 | 87.8 | 73.7 |
| 2048 | 128 | 32 | 128 | 4534.8 | 67.0 | 14.9 | 29.7 | 313.7 |
| 4096 | 128 | 1 | 4 | 578.2 | 11.5 | 87.2 | 87.0 | 62.9 |
| 4096 | 128 | 32 | 128 | 6225.0 | 127.8 | 7.8 | 27.8 | 182.2 |
| 8192 | 128 | 1 | 2 | 1185.0 | 11.7 | 85.4 | 85.2 | 47.9 |
| 8192 | 128 | 31 | 62 | 12485.7 | 225.8 | 4.4 | 25.7 | 96.1 |
| 8192 | 1024 | 1 | 2 | 1189.2 | 11.8 | 84.9 | 84.9 | 77.3 |
| 8192 | 1024 | 28 | 56 | 11620.7 | 57.8 | 17.3 | 26.9 | 404.7 |
| 10000 | 1024 | 1 | 2 | 1519.8 | 11.9 | 83.7 | 83.7 | 74.5 |
| 10000 | 1024 | 23 | 46 | 12789.5 | 57.6 | 17.4 | 28.1 | 328.1 |
| 16384 | 128 | 1 | 2 | 2535.3 | 12.2 | 82.2 | 81.7 | 31.4 |
| 16384 | 128 | 15 | 30 | 14173.7 | 218.3 | 4.6 | 29.4 | 45.7 |
| 32768 | 128 | 1 | 1 | 5679.0 | 12.2 | 81.7 | 80.6 | 17.7 |
| 32768 | 128 | 7 | 7 | 22652.5 | 154.3 | 6.5 | 45.6 | 21.1 |
| 65536 | 128 | 1 | 1 | 13779.0 | 12.9 | 77.7 | 75.7 | 8.3 |
| 65536 | 128 | 3 | 3 | 27443.0 | 126.4 | 7.9 | 51.3 | 8.8 |
| 131072 | 128 | 1 | 1 | 36994.8 | 14.4 | 69.7 | 67.0 | 3.3 |
| 196608 | 128 | 1 | 1 | 70081.0 | 15.6 | 64.2 | 60.7 | 1.8 |
| 262016 | 128 | 1 | 1 | 112199.5 | 16.9 | 59.3 | 55.3 | 1.1 |
Run 2026-11-17 15:06–15:33 (KST), tt-inference-server 0.26.1 --workflow benchmarks against tt-model serve --profile batch; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 14 of 25 points (chips 0, 1, 2, 3; up to 87% of decode time: 1024/128 c1, 1024/128 c32, 2048/128 c1, 2048/128 c32, 4096/128 c32, 8192/128 c31, 8192/1024 c1, 8192/1024 c28, 10000/1024 c1, 10000/1024 c23, 16384/128 c15, 32768/128 c1, 32768/128 c7, 65536/128 c3).
¹ Decode t/s/u is 1000 / the median gap between consecutive streamed tokens (the point file's median_itl_ms): the
decode rate without the pauses that other users' prefills add to TPOT. At concurrency 1 there are no such pauses;
there the two columns differ by at most 7.2% (262,016/128: 55.3 vs 59.3). single-user's tables have no such column,
because speculative decoding streams several tokens at a time.
batch: most concurrent users admitted at once per context length (vLLM KV accounting)
| ISL | OSL | Concurrency (target)² | Achieved concurrency² | Num Requests | TTFT (ms)² | TPOT (ms) | Tput User (t/s/u = 1000/TPOT) | Decode t/s/u (1000/median ITL)¹ | Tput Output (t/s) |
|---|---|---|---|---|---|---|---|---|---|
| 16384 | 128 | 17 | 16 | 34 | 14332.0 | 256.3 | 3.9 | 28.2 | 46.3 |
| 32768 | 128 | 10 | 10 | 10 | 30997.0 | 232.4 | 4.3 | 28.2 | 21.1 |
| 65536 | 128 | 6 | 6 | 6 | 47919.5 | 293.0 | 3.4 | 37.8 | 9.0 |
| 131072 | 128 | 3 | 3 | 3 | 73832.9 | 313.2 | 3.2 | 39.9 | 3.4 |
Run 2026-11-17 17:02–17:06 (KST, start of each point), the vllm bench serve invocation tt-inference-server 0.26.1 issues for its points, against tt-model serve --profile batch; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 4 of 4 points (chips 0, 1, 2, 3; up to 87% of decode time: 16384/128 c17, 32768/128 c10, 65536/128 c6, 131072/128 c3).
² tt-inference-server's own points cap concurrency at 262,144 / (ISL + OSL), one maximum context in total. These points
ask instead for the most users vLLM admits at once: the target is vLLM's runtime admission rule, calibrated
on the batch session above (est.). vLLM's KV accounting reports a capacity of 414,365 tokens: it
charges batch's five sliding-window layer groups against the same pool of 4,320 blocks of 128 tokens as the
full-attention layers, so fewer users fit than the full-attention blocks alone would hold. By the same
accounting a second 262,016-token request does not fit while one runs (the table above has that point at one
user). "Achieved concurrency" is the most requests running together (vLLM's
10 s stats): at 16,384 tokens 16 of the 17 ran together, and every other point ran at its target. The users' prefills run one after another, so later users' TTFT includes the earlier prefills.
¹ As in the table above.
Serve profiles
One image serves every profile below; pick one with --profile.
| profile | hardware | mesh | max_num_seqs | max_model_len |
|---|---|---|---|---|
single-user (default) |
p300x2 | P300x2 | 1 | 262144 |
batch |
p300x2 | P300x2 | 32 | 262144 |
Provenance
The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | 6e7775484d0115844b0c33125d0fb4f6ae7996c5 |
| vLLM | v0.31.0 |
| vllm-tt-plugin | ca7a18246e4111629ed6cb4a7d1192002749fccf |
code/ digest |
e08ccfc06498335e (sha256, first 16 hex digits) |
| built | 2026-11-17T08:23:19+00:00 by tt-model 0.1.0 |