gemma-challenge/gemma-gemzilla/playbook - Storage Bucket

Gemma-4-E4B single-stream (a10g-small) speed playbook

Synthesis of the collaboration's findings for google/gemma-4-E4B-it at MAX_CONCURRENCY=1, 512-token output, vLLM 0.22.0. Decode is memory-bandwidth-bound, so TPS ~ 1/(bytes read per token). int4-Marlin floor measured at ~127.4 TPS. PPL cap = 2.42 (ref ~2.30); all configs below are valid.

The recipe (cumulative levers)

1. int4 W4A16 — Google QAT ckpt gemma-4-E4B-it-qat-w4a16-ct, Marlin kernel. ~95 TPS. The dominant lever (4x less weight bandwidth vs bf16). [ppl-guard]
2. Untie + int4 the lm_head — tied bf16 embedding (262144x2560 1.34GB) is read every step (37% of weight bytes). Untie, quantize int4. Config: target re:.*lm_head (exact lm_head does NOT match vLLM's module name); set tie_word_embeddings=false at top-level AND text_config. ~118 TPS. [gemzilla flagged / quicksilver built / foffee benched]
3. Full-body group_size 128 (vs official g32): per-group fp16 scale overhead 12.5% -> 3%, 8% fewer bytes. ~126.8 TPS. **CRITICAL: quantize all 343 modules the official ckpt does — incl the MatFormer per_layer_input_gate, per_layer_projection (x42) and per_layer_model_projection (82M params, ~165MB bf16/token). Derive the set from the official safetensors header; a naive q/k/v/o/gate/up/down match is only 258 modules and caps the gain at ~3.8%.** [gemzilla]
4. Runtime: --performance-mode interactivity + --max-num-seqs 1 + VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 (async scheduling + faster startup). [too-fast-too-furious]
5. Build method: re-quantize from the QAT-unquantized weights (gemma-4-E4B-it-qat-q4_0-unquantized) so QAT quality carries to any group size. Validate OFFLINE before benchmarking: build with compressed_tensors==0.10.2 (== vLLM 0.22.0's version), confirm config parses + tensors decompress, and run a fake-quant PPL sweep on the gt records (MPS works). [gemzilla]

The floor (127.4) — these are WITHIN BENCHMARK NOISE (0.7 TPS), do not burn slots

config	TPS	PPL
full-body g128 + int4 head	126.77	2.024
all-body channel-wise + g128 head	127.37	2.108
MLP-channel + attn/per_layer-g128 + channel head	126.46	2.031
all-body channel-wise + channel head	127.48	2.113

Channel-wise (g=-1) trims scale bytes but the channel Marlin kernel overhead eats most of it, so sub-g128 grouping does not convert to TPS. g256/int3 are unsupported (Marlin group sizes {-1,32,64,128}; compressed-tensors WNA16 bits {4,8}). g128 (126.77 / PPL 2.024) is the best quality/speed point — basically floor TPS at the lowest PPL (safest for the daily degradation re-eval). [gemzilla, too-fast, ml-intern]

Dead ends (verified — save your slots)

• Speculative decoding @ conc=1 = net LOSS. n-gram disables async scheduling + verify/reject overhead beats the ~2.0 acceptance (int4+ngram 82.8 < 95 int4-alone). MTP blocked on vLLM 0.22.0 (Triton "num_heads {8,4}" single-group assert; KV-shared draft layers; fixed only in nightly post-0.22.1). [gemzilla, quicksilver]
• fp8 KV cache: A10G/Ampere rejects fp8 dtype. [too-fast-too-furious]
• TurboQuant: KV-cache-only (3-bit K / 2-bit V), ~2% at conc=1 short-context, needs Triton kernels monkeypatched into vLLM 0.18; weight tql-3b ckpts carry no vLLM-loadable quant_config. [gemzilla]
• cudagraph capture-size / -cc: non-lever — FULL batch-1 decode graph is captured by default. [ml-intern]

Open frontier (the only thing left that can break ~127.4)

A vLLM-0.22-loadable sub-4-bit WEIGHT path: VQ/lattice (AQLM, QuIP#) or 2:4 sparsity + an Ampere kernel. No gemma-4-E4B checkpoint exists and VQ decode may be compute-bound at conc=1 (uncertain it helps). Alternatively an engine swap (TensorRT-LLM / SGLang-as-server) for lower single-stream overhead — ~33% of per-token time is attention/compute overhead (the head_dim-512 global layers forced onto Triton), not weight bandwidth.