axteam/gemma-4-31B-FP8-dynamic

gemma-4-31B · FP8 (dynamic)

8-bit quantization of google/gemma-4-31B (commit 5bbc2fb), produced for efficient vLLM serving. Quantized by axteam.

Base model	google/gemma-4-31B (commit 5bbc2fb)
Method	FP8 (dynamic)
Format	compressed-tensors FP8_DYNAMIC
Weights / activations	W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations
Architecture	Gemma 4 · Dense
Size on disk	~31.0 GB
Calibration	None (data-free static weight scales)
License	Gemma 4 / Apache-2.0

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations, stored as compressed-tensors FP8_DYNAMIC.
• Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
• Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Evaluation

Quantization fidelity was checked against the BF16 original using perplexity and top-1 next-token agreement on an internal mixed English/Korean set. In our runs the 8-bit variants (AWQ INT8 / FP8) stay closest to the original, while the 4-bit variants (AWQ INT4 / NVFP4) trade a little quality for roughly half the footprint.

Usage (vLLM)

vllm serve axteam/gemma-4-31B-FP8-dynamic

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-31B-FP8-dynamic")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Thinking mode: add the <|think|> token at the start of the system prompt to enable step-by-step reasoning.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-31B-FP8-dynamic --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

---

한국어 요약

google/gemma-4-31B 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — FP8 E4M3. 가중치는 출력 채널마다 스케일 하나(정적·보정 불필요), 활성화는 추론 시 토큰마다 동적 FP8. 형식은 compressed-tensors FP8_DYNAMIC.
• BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
• 토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
• 품질 — BF16 원본 대비 perplexity·top-1 일치도로 검증(영/한 혼합셋). 8비트(AWQ INT8·FP8)가 원본에 가장 가깝고, 4비트(AWQ INT4·NVFP4)는 용량을 절반으로 줄이는 대신 품질을 조금 양보.
• 사용 — vllm serve axteam/gemma-4-31B-FP8-dynamic. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
• KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선 vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
• 라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.