gemma4-31b-it-dflash

google/gemma-4-31B-it on a QuietBox 2 (P300x2, 4 Blackhole chips) with vLLM. The default profile serves one user on a resident decode engine (one persistent program per chip per decode step) with DFlash2 speculative decoding, up to 262K context. batch serves up to 32 users on a resident batch engine that keeps per-user decode fast at long context, up to 262K. Both profiles use a paged-bucket prefill: a fixed set of prefill programs built at start-up, so device memory no longer grows with the number of distinct prompt lengths served.

Runs on p300x2 — see the serve profiles below.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart

tt-model pull  changh95/gemma4-31b-it-dflash --with-weights
tt-model serve changh95/gemma4-31b-it-dflash

pull --with-weights downloads the Docker image and the google/gemma-4-31B-it weights at 842da3794eaa0b77d5f08bae87a17459d91ff475 (into your HF cache; they are not in the image). serve starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy); the first start compiles kernels for your device, which takes several minutes, and the server is ready when it logs Application startup complete.

At a glance

  • Weights: google/gemma-4-31B-it @ 842da37; drafter RedHatAI/gemma-4-31B-it-speculator.dflash2 @ b0cd01b
  • Hardware: P300x2 (QuietBox 2), one (1, 4) Blackhole mesh, tensor-parallel over 4 chips
  • Profiles: single-user (default), batch
  • Context: 262,144 tokens on every profile
  • Weights on device: BF4 (bfloat4_b) projections, BF8 LM head and KV cache, as validated
  • Thinking is on by default. Turn it off per request with "chat_template_kwargs": {"enable_thinking": false}.
  • Default sampling: the model's generation config (temperature 1.0, top_p 0.95) with top_k 20. Pass "temperature": 0 for greedy.
  • single-user speculates on greedy and on sampled requests (temperature, top_p, top_k). Penalties and top_k above 32 are served without speculation. Speculative output can differ from plain decoding at near-ties.
  • single-user refuses (HTTP 400) what its speculative path cannot apply: logprobs, structured outputs (response_format, guided decoding), bad_words, allowed_token_ids, min_tokens, logit_bias, min_p. batch serves logprobs (on the host-sampled step; see Known limitations).
  • Tool calling: single-user parses tool calls (vLLM's gemma4 parser, tool_choice: "auto"). batch has no tool calling.
  • batch (update 3): the resident batch engine. It picks its decode width by the number of live users (R = 1 / 8 /
    1. and does not speculate.

Update 2: paged-bucket prefill

  • Both profiles now build their prefill programs for a fixed set of length buckets at start-up. Update 1 compiled a new program set for every distinct prompt length and kept it for the life of the server, about 0.24 MB of device memory per chip per new length. Update 2's program set is bounded by the fixed buckets.
  • Decoding is unchanged. In the gating runs, update 2's outputs matched update 1's on the same prompts.
  • Start-up cost, measured with the kernels already compiled: about 5 s more for single-user, which also pre-builds the drafter's seed programs (about 6.5 MB per chip). The extra compile time on a first start has not been measured yet.
  • A prompt longer than one 8,192-token chunk builds a few programs the first time a new continuation length class is seen (lazy, on every profile). single-user ignores GEMMA4_PREFILL_WARM_CONT=1 (its continuation programs are always built on first use, and its drafter seed pre-warm is always on); batch refuses to start with it set.
  • Measured first-sight cost of a new continuation length class, on a fresh kernel cache: about +1.05 s TTFT on single-user (kernel compilation, once per host per class); about 10 ms once the kernels are on disk. Not measured on batch.

Update 3: the resident batch engine (batch)

--profile batch now selects the resident batch engine. The previous batch-b32 profile was removed; the update-2 revision keeps it.

  • Start-up, measured in update 3.1's proof (in the package container, the same code): 120 s to ready on a warm start. The first batch start on a host that had already run single-user once took 241 s, including the one-time conversion of batch's own copy of the engine weights (converted in 55 s; 22.8 GB more disk). A fresh host that starts batch first also compiles the shared kernels: that start is not measured. single-user's weight copy is untouched.
  • batch serves logprobs, repetition / presence / frequency penalties and logit_bias on its host-sampled step, each alone and alongside 31 greedy users (checked in this release's proof); see Known limitations.
  • One long request among short ones: 32 users with 1K-token prompts (512 output tokens) had a TPOT of 35.6 ms; with one of them replaced by a 259,000-token request, its prefill already done, the 31 short users' TPOT was 42.5 ms (2026-11-15, the same engine before the paged-bucket prefill was added; chip clocks were below 1,323 MHz for 4–19% of the 32-user windows).
  • Reliability: 0 hangs in 491,520 long-context launches (≤ 6.1e-6 per launch, 95%), in a 6.5 h soak of batch's engine at 262K context.

Real prompts. vllm bench serve (tt-inference-server's client) with the real-prompt builder's chat prompts: a coding, writing or math task after a real context, cut so the chat template renders it to exactly the stated length; thinking on (the server default), greedy, 256 output tokens, warm-ups = concurrency. Times are as measured; points where chip clocks dropped during decode carry a footnote with the per-chip share of decode time below 1,323 MHz.

ISL Concurrency TPOT (ms) TTFT (ms)
128 1 10.2 64
128 8 14.3 392
128 32 26.8 1,465
1,024 1 11.4 145
1,024 8 16.7[1] 1,054
1,024 32 38.0[2] 2,905
8,192 1 11.7[3] 1,181
32,768 1 12.4[4] 5,635
  • [1] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 10.48 / 10.48 / 0.04 / 10.48. Times as measured, not corrected.
  • [2] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 14.17 / 17.48 / 0 / 17.48. Times as measured, not corrected.
  • [3] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 0 / 17.64 / 26.46 / 17.64. Times as measured, not corrected.
  • [4] Chip clocks dipped during this point's decode. Share of decode time below 1,323 MHz, chips 0 / 1 / 2 / 3 (%): 14.95 / 34.24 / 28.94 / 24.11. Times as measured, not corrected.
  • The clock footnotes cover the decode windows (TPOT) only.

Measured 2026-11-16 09:28–09:31 (KST), outside the package container on the same files (byte-identical engine and adapter code); this release's proof measures the packaged image at the same points.

Which profile

  • One user: single-user.
  • 2 to 32 concurrent users, or several users at long context: batch.

Get the drafter (single-user)

hf download RedHatAI/gemma-4-31B-it-speculator.dflash2 --revision b0cd01b4918aae6b6388272950c4f9851734674b

Serve

tt-model serve changh95/gemma4-31b-it-dflash                    # default profile: single-user
tt-model serve changh95/gemma4-31b-it-dflash --profile batch
tt-model profiles changh95/gemma4-31b-it-dflash
  • First start on a host (single-user): about 12.5 min to ready. It converts the resident engine's weights once (about 70 s: 22 GB from the 59 GB checkpoint) and compiles kernels once (about 9.5 min).
  • Later starts: about 2.5 min, including a sha256 check of the converted weights (seconds when they are in the page cache).
  • These start times were measured on update 1. Update 2 adds its prefill warm-up (see above); batch's start-up is in the update 3 section.
  • Disk: about 31 GB under ~/.cache/tt-model/gemma4-31b-it-dflash (22 GB converted weights, 9 GB compiled kernels), plus 22 GB for batch's own weight copy once it has started. A package update that changes an engine converts that engine's copy again; single-user's copy is unchanged by update 3.

How to query

curl http://localhost:20000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "google/gemma-4-31B-it",
  "messages": [{"role": "user", "content": "Write a haiku about Blackhole chips."}],
  "max_tokens": 256, "temperature": 0,
  "chat_template_kwargs": {"enable_thinking": false}
}'

Known limitations

  • Code-generation quality (BF4 weights). On the same evaluation harness, run through update 1's batch decode path, a build of this model with BF8 weights scores higher on code generation:
    • MBPP: 0.566 (this build) vs 0.658, a significant gap (paired McNemar test, Holm-adjusted p ≈ 1e-6).
    • HumanEval: 0.835 vs 0.884, not significant.
    • About half of the MBPP gap is answers that never close their code block, mostly because they run long or repeat until the 512-token limit.
    • Long thinking traces repeat more often: 12.8% vs 8.8% are degenerate.
    • GSM8K, IFEval, MMLU-Pro, MATH-500 and RULER (8K) are within noise.
  • Host-sampled requests in batch. A request that needs CPU sampling (repetition, presence or frequency penalties, logprobs, or top_k above the device limit of 32) moves sampling for every concurrent request to the CPU while it is active.
    • At 32 users a decode step takes 277.5 ms while one of them carries a repetition penalty (the other 31 default-sampled), against 26.9 ms when all 32 are greedy (update 3.1's proof, same code, with OMP_NUM_THREADS=8, which this package sets).
    • Greedy requests and the default device sampling are unaffected.
    • The full fix belongs in the vLLM TT plugin.
  • batch: greedy text depends on load. The engine changes its decode width (R = 1 / 8 / 32) as users join and leave, and the same greedy prompt can produce different text depending on how many other requests are running. At a fixed width, greedy text was bitwise stable under co-tenant churn for prompts that prefill in one step. A prompt that is chunked alongside others can see small differences.
  • batch: prefix caching is off (as on every profile): repeated prefixes are recomputed.
  • Device memory. Lowest free DRAM per chip measured on batch: 2.85 GB, briefly, right after a burst of 1K-prompt prefills at up to 32 users with a 200K-token request live (a 6.5 h, 491K-launch soak). At steady state about 6.3 GB, flat over the soak.
  • Thermal clock dips. On a QuietBox 2 chip 1 can lower its clock after about 20 s of sustained fast decode, raising per-token time by up to about 1.3–1.45× while it lasts (measured on earlier builds). Every table on this card notes the clock dips during its run.
  • batch: host decode guard. The engine refuses a decode mix whose conservative attention bound exceeds 40 units per core. With its KV pool (4,320 blocks, about 553K positions) no reachable mix comes near it (at most 25 of 40); it would matter only with a pool about 4.5 times larger (above 19,648 blocks).
  • Update 2 evidence scope. The paged-bucket prefill's runtime guard and its gating runs check program-cache growth and outputs; they did not record device buffer addresses. The drafter seed pre-warm's first-start (cold kernel cache) time is not measured.

Results (tt-inference-server --workflow benchmarks)

The first three tables were measured in one session on 2026-11-17 on update 3.1's image (7b6dd89c97bd), whose code is identical to this release's (update 3.2 changes only the manifest: this card and the profile list): each profile in its own tt-model serve --profile … container on port 8000, with the same tt-inference-server invocations, specs and points as update 1's tables of 2026-11-11, which remain in the update-2 revision on the Hub. The fourth was measured in a second session on 2026-11-17, on the same image.

single-user

ISL OSL Concurrency Num Requests TTFT (ms) TPOT (ms) Tput User (t/s/u = 1000/TPOT) Tput Output (t/s)
128 128 1 8 58.4 4.5 224.6 205.0
128 1024 1 4 59.1 3.8 266.0 262.2
1024 128 1 4 137.7 8.7 115.1 103.1
2048 128 1 4 281.0 8.9 112.1 90.5
4096 128 1 4 565.8 8.8 113.0 75.7
8192 128 1 2 1168.9 7.5 132.9 60.2
8192 1024 1 2 1171.5 8.7 114.6 101.4
10000 1024 1 2 1483.6 3.9 258.5 188.2
16384 128 1 2 2500.8 11.9 83.9 31.9
32768 128 1 1 5617.2 8.5 117.5 19.1
65536 128 1 1 13642.1 4.9 205.2 9.0
131072 128 1 1 36924.4 5.0 201.8 3.4
196608 128 1 1 69572.7 5.1 195.8 1.8
262016 128 1 1 111630.0 5.7 175.0 1.1

Run 2026-11-17 15:42–15:50 (KST), tt-inference-server 0.26.1 --workflow benchmarks against tt-model serve --profile single-user; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 7 of 14 points (chips 0, 1, 2, 3; up to 36% of decode time: 1024/128 c1, 2048/128 c1, 4096/128 c1, 8192/128 c1, 8192/1024 c1, 10000/1024 c1, 16384/128 c1). Greedy continuations of random prompts repeat and draft unusually well, so these rates are above what real text gets; the realistic-prompt numbers below are the better guide.

single-user, realistic prompts (tt-inference-server --spec-decode)

Prompts (SPEED-Bench) Concurrency Num Requests TTFT (ms) TPOT (ms) Tput User (t/s/u = 1000/TPOT) Tput Output (t/s) Tokens per verify (of 8)
coding 1 80 121.6 4.9 204.3 190.2 4.60
throughput_32k 1 32 6641.3 11.1 89.8 69.5 2.64

Run 2026-11-17 15:58–16:28 (KST), tt-inference-server 0.26.1 --workflow benchmarks --spec-decode against tt-model serve --profile single-user: SPEED-Bench prompts (AIPerf; the requests carry no sampling fields, so the server's defaults apply: temperature 1.0, top_p 0.95, top_k 20, thinking on). Each answer runs to its natural end (throughput sets capped at 8,192 tokens). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz for coding 0.1% of decode time; throughput_32k 0.5% of decode time (76% of prefill).

batch (the resident batch engine)

ISL OSL Concurrency Num Requests TTFT (ms) TPOT (ms) Tput User (t/s/u = 1000/TPOT) Decode t/s/u (1000/median ITL)¹ Tput Output (t/s)
128 128 1 8 65.1 10.1 99.3 98.9 95.2
128 128 32 256 1471.8 26.9 37.2 37.2 838.6
128 1024 1 4 64.9 10.8 92.5 89.6 92.1
128 1024 32 128 1464.0 29.3 34.1 34.5 1041.8
1024 128 1 4 148.1 11.4 88.0 88.0 80.4
1024 128 32 128 3026.1 41.8 23.9 30.6 491.4
2048 128 1 4 291.6 11.4 87.8 87.8 73.7
2048 128 32 128 4534.8 67.0 14.9 29.7 313.7
4096 128 1 4 578.2 11.5 87.2 87.0 62.9
4096 128 32 128 6225.0 127.8 7.8 27.8 182.2
8192 128 1 2 1185.0 11.7 85.4 85.2 47.9
8192 128 31 62 12485.7 225.8 4.4 25.7 96.1
8192 1024 1 2 1189.2 11.8 84.9 84.9 77.3
8192 1024 28 56 11620.7 57.8 17.3 26.9 404.7
10000 1024 1 2 1519.8 11.9 83.7 83.7 74.5
10000 1024 23 46 12789.5 57.6 17.4 28.1 328.1
16384 128 1 2 2535.3 12.2 82.2 81.7 31.4
16384 128 15 30 14173.7 218.3 4.6 29.4 45.7
32768 128 1 1 5679.0 12.2 81.7 80.6 17.7
32768 128 7 7 22652.5 154.3 6.5 45.6 21.1
65536 128 1 1 13779.0 12.9 77.7 75.7 8.3
65536 128 3 3 27443.0 126.4 7.9 51.3 8.8
131072 128 1 1 36994.8 14.4 69.7 67.0 3.3
196608 128 1 1 70081.0 15.6 64.2 60.7 1.8
262016 128 1 1 112199.5 16.9 59.3 55.3 1.1

Run 2026-11-17 15:06–15:33 (KST), tt-inference-server 0.26.1 --workflow benchmarks against tt-model serve --profile batch; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 14 of 25 points (chips 0, 1, 2, 3; up to 87% of decode time: 1024/128 c1, 1024/128 c32, 2048/128 c1, 2048/128 c32, 4096/128 c32, 8192/128 c31, 8192/1024 c1, 8192/1024 c28, 10000/1024 c1, 10000/1024 c23, 16384/128 c15, 32768/128 c1, 32768/128 c7, 65536/128 c3).

¹ Decode t/s/u is 1000 / the median gap between consecutive streamed tokens (the point file's median_itl_ms): the decode rate without the pauses that other users' prefills add to TPOT. At concurrency 1 there are no such pauses; there the two columns differ by at most 7.2% (262,016/128: 55.3 vs 59.3). single-user's tables have no such column, because speculative decoding streams several tokens at a time.

batch: most concurrent users admitted at once per context length (vLLM KV accounting)

ISL OSL Concurrency (target)² Achieved concurrency² Num Requests TTFT (ms)² TPOT (ms) Tput User (t/s/u = 1000/TPOT) Decode t/s/u (1000/median ITL)¹ Tput Output (t/s)
16384 128 17 16 34 14332.0 256.3 3.9 28.2 46.3
32768 128 10 10 10 30997.0 232.4 4.3 28.2 21.1
65536 128 6 6 6 47919.5 293.0 3.4 37.8 9.0
131072 128 3 3 3 73832.9 313.2 3.2 39.9 3.4

Run 2026-11-17 17:02–17:06 (KST, start of each point), the vllm bench serve invocation tt-inference-server 0.26.1 issues for its points, against tt-model serve --profile batch; random prompts, greedy (temperature 0), thinking on (server default). One of the QuietBox's two fans was dead during the run (not yet replaced); a chip's clock was below 1,323 MHz during decode at 4 of 4 points (chips 0, 1, 2, 3; up to 87% of decode time: 16384/128 c17, 32768/128 c10, 65536/128 c6, 131072/128 c3).

² tt-inference-server's own points cap concurrency at 262,144 / (ISL + OSL), one maximum context in total. These points ask instead for the most users vLLM admits at once: the target is vLLM's runtime admission rule, calibrated on the batch session above (est.). vLLM's KV accounting reports a capacity of 414,365 tokens: it charges batch's five sliding-window layer groups against the same pool of 4,320 blocks of 128 tokens as the full-attention layers, so fewer users fit than the full-attention blocks alone would hold. By the same accounting a second 262,016-token request does not fit while one runs (the table above has that point at one user). "Achieved concurrency" is the most requests running together (vLLM's 10 s stats): at 16,384 tokens 16 of the 17 ran together, and every other point ran at its target. The users' prefills run one after another, so later users' TTFT includes the earlier prefills. ¹ As in the table above.

Serve profiles

One image serves every profile below; pick one with --profile.

profile hardware mesh max_num_seqs max_model_len
single-user (default) p300x2 P300x2 1 262144
batch p300x2 P300x2 32 262144

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component built from
tt-metal 6e7775484d0115844b0c33125d0fb4f6ae7996c5
vLLM v0.31.0
vllm-tt-plugin ca7a18246e4111629ed6cb4a7d1192002749fccf
code/ digest e08ccfc06498335e (sha256, first 16 hex digits)
built 2026-11-17T08:23:19+00:00 by tt-model 0.1.0
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support