gemma4-spark-testbench

A test bench, not a model. This repository documents a series of hands-on experiments aimed at making google/gemma-4-26B-A4B-it decode faster on bandwidth-limited hardware, and at understanding how the model behaves internally under different serving and probing strategies. Everything here is measurement: what we tried, how we tried it, what worked, and what did not.

There are no model weights in this repository. Scripts, result data, patches, and write-ups only.

Disclaimer — preliminary results. Nothing in this repository should be treated as conclusive. All results come from a single machine, a single software stack, and small sample sizes, and any experiment here may contain undetected errors in methodology, instrumentation, or interpretation. Further research and independent replication are needed to determine whether any given test — including the ones reported as successes — is wrong. The scripts are included precisely so the results can be checked. Treat everything here as a starting point, not a finding of record.

Test system

Component	Detail
Hardware	NVIDIA DGX Spark (GB10, Blackwell SM121), 121 GB unified LPDDR5x
Measured memory bandwidth	~232 GB/s sustained (roofline for all decode numbers below)
Serving stack	vLLM 0.26.0
Analysis stack	PyTorch 2.12 (cu130), transformers 5.10–5.14
Model	gemma-4-26B-A4B-it (30 layers, 128 experts / top-8, 262,144 vocab, tied embeddings)

Batch-1 decode on this class of hardware is memory-bandwidth-bound: tokens/s ≈ bandwidth ÷ bytes-read-per-token. Most of what follows is an attempt to shrink the numerator's per-token byte cost or amortize it across more tokens.

Findings index

Doc	Topic	Headline result
01 — Bandwidth baseline	Byte budgets and ceilings	Stock NVFP4 52.5 tok/s vs ~72 ceiling; per-token reads 3.23 GB
02 — FP4 lm_head recipe	Quantizing the output head (published NVFP4 quants keep it bf16)	52.5 → 82.6 tok/s (+57%), tool-use intact; three non-obvious config fixes required
03 — Speculative decoding	Official MTP, EAGLE-3, policy simulation, zero-cost drafting	Acceptance is terrain-dependent (code ≥0.79, chat word-boundaries ~0.54); zero-training leftover drafting falsified
04 — Native FP4×FP4	W4A4 attempts on SM121	Native path numerically wrong on this stack; outlier-permutation rescue falsified; marlin remains the working lane
05 — Inside the model	Logit-lens depth map, future-token preview, output-head clustering	Prediction surfaces only in the last 1–2 layers; next-next token broadcast at rank ~100 of 262k; clustered-head retrofit falsified
06 — Limitations	Scope and caveats	Small n, one box, one stack — all verdicts are scoped

What this bench is for

1. Speed on small bandwidth. The model's published serving numbers assume datacenter bandwidth. On a 232 GB/s box every byte matters; the experiments in docs 01–04 measure which byte-reduction and amortization levers actually pay on this hardware, with full A/B data.
2. Model behavior under criteria. Docs 05 probes how the model forms its predictions: at what depth the next token condenses, how much future-token information the current distribution carries, and whether the output vocabulary's geometry can be exploited. These are reusable instruments — every script runs against a local HF checkpoint with no serving stack.

Negative results are results

Several experiments here are falsifications, reported with the same care as the wins: the native W4A4 path on this stack, an outlier-permutation rescue, training-free early-exit drafting, zero-cost leftover drafting, and a clustered output head retrofitted onto the frozen checkpoint. Each report includes the mechanism of failure, not just the outcome, since the mechanism is usually the transferable part.

Repository layout

docs/        clean write-ups (start here)
scripts/     runnable instruments and recipes (paths are local; adjust MODEL_DIR)
data/        raw result JSON/NPZ from the runs cited in docs
deep-dives/  longer working notes behind docs 01–04
patches/     vLLM patches required to reproduce (each has its own README)

Upstream reports filed from this work

• vLLM: Gemma-4 MTP drafter fails to boot with --speculative-config (embedding-width sharing skip) — patch in patches/vllm_gemma4_mtp_embed_share/, reported upstream in the linked PR discussion in that patch's README.
• Observed (unfiled at time of writing): vLLM 0.26.0 cannot serve the BF16 gemma-4-26B-A4B base coherently at any context length, while HF transformers serves it correctly — details in doc 04.

AI assistance disclosure

The experiments in this repository were designed, executed, and documented with substantial AI assistance: Anthropic's Claude carried out experiment design, instrumentation, runs, and the write-ups in docs/; one working note in deep-dives/ was produced by GLM-5.2. Research direction, hypotheses, review, and all publication decisions are the repository owner's. Result data in data/ is raw instrument output, reported as measured.

Status

Private staging. Numbers in docs are from runs of n = 191–231 positions (behavioral probes) or standard serving benchmarks (A/B, 3-run medians); see doc 06 before quoting anything.