Gemma 4 31B IT — MXFP4 Quantized

google/gemma-4-31B-it quantized to MXFP4 (W4A16) using GPTQ with special techniques that preserve Korean and English performance simultaneously.

Model Details

Property	Value
Base model	`google/gemma-4-31B-it` (BF16, 62.5 GB)
Quantization format	MXFP4 W4A16 (compressed-tensors)
Disk size	19.5 GB (69% reduction from BF16)
VRAM (model weights)	~19 GB
Calibration data	BCCard on-policy 600k

Serving

vLLM (recommended)

# Standard environment (CUDA 12.9+):
vllm serve gemma-4-31B-it-MXFP4 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

# Blackwell with CUDA < 12.9:
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve gemma-4-31B-it-MXFP4 \
  --port 8000 \
  --linear-backend marlin \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Blackwell Compatibility

On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:

Setting	Purpose
`VLLM_USE_FLASHINFER_SAMPLER=0`	Disables FlashInfer sampler (uses native PyTorch)
`--linear-backend marlin`	Uses pre-compiled MXFP4 GEMM kernels (no JIT)

Component matrix after workarounds:

Component	Backend	JIT Required
MXFP4 GEMM	Marlin (pre-compiled)	No
Attention	TRITON_ATTN	No (Triton JIT, not nvcc)
Sampling	Native PyTorch	No

Evaluation

Quality Benchmarks

Task	MXFP4 (this model)	BF16 baseline	Recovery
HellaSwag (English)	89.5%	91.0%	98.4%
KoBEST-HellaSwag	73.0%	76.5%	95.4%
KoBEST-COPA	99.0%	99.0%	100.0%
KoBEST-BoolQ	95.0%	96.5%	98.4%
KoBEST-SentiNeg	99.0%	99.0%	100.0%
KoBEST-WiC	98.5%	99.0%	99.5%
KMMLU (5 subjects)	76.0%	78.0%	97.4%
BC-Finance QA (char-F1)	0.387	0.386	100.3%
Average recovery			98.7%
Korean Generation	476 tok, rep 0.99, Hangul 59.2%	482 tok, rep 0.99, Hangul 57.9%	—
English Generation	424 tok, rep 0.99	421 tok, rep 1.00	—

Korean Knowledge (KMMLU) — Reference Points

Published scores from other Korean(-capable) models, for orientation. These are NOT apples-to-apples: vendors use the full 45-subject KMMLU (or the harder KMMLU-Redux subset) with their own prompting/shot settings, while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample from the harness above.

Model	Params	Disk	Score	Benchmark (setting/source)
A.X 4.0 (SKT)	72B	~144 GB	78.3	KMMLU, vendor
Gemma 4 31B IT (BF16 baseline)	31B	62.5 GB	78.0	KMMLU 5-subject sample, 0-shot chat MCQ (this harness)
This model (Gemma 4 31B MXFP4, 4-bit)	31B	19.5 GB	76.0	KMMLU 5-subject sample, 0-shot chat MCQ (this harness)
EXAONE 4.0 (LG)	32B	~64 GB	75.2	KMMLU, as reported in the HyperCLOVA X 32B Think report
GPT-4o (OpenAI)	—	— (API)	72.5	KMMLU, as reported by SKT A.X
HyperCLOVA X 32B Think (NAVER)	32B	~64 GB	71.3	KMMLU, vendor report
A.X 3.1 (SKT)	34B	~68 GB	69.2	KMMLU, vendor
A.X 3.1 Light (SKT)	7B	~14 GB	61.7	KMMLU, vendor
Gemma 3 27B IT (Google)	27B	~54 GB	54.0	KMMLU-Redux, KMMLU-Redux/Pro paper

Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.

Directional takeaway: on this 5-subject sample the BF16 gemma-4-31B-it baseline (78.0) sits on par with A.X 4.0 72B (78.3), and the 4-bit MXFP4 model retains that tier — landing above EXAONE 4.0 32B and GPT-4o — at a 19.5 GB footprint. Against its own BF16 baseline under the identical harness, the model averages 98.7% recovery across all scored tasks.

Note: Perplexity cannot be measured with MXFP4 + Marlin backend due to NaN in prompt_logprobs. Text generation and classification tasks work normally.

Standard Benchmarks (lm-eval-harness)

Benchmark	BF16 Baseline	RedHatAI FP8	Recovery%	This Model (MXFP4)
GSM8K	96.11	96.22	100.1%	TBD
MMLU-Pro	87.05	87.07	100.0%	TBD
IFEval	94.58	94.89	100.3%	TBD

RedHatAI FP8 reference: RedHatAI/gemma-4-31B-it-FP8-dynamic

Compression Comparison

Format	Disk Size	vs BF16	Notes
BF16 (original)	62.5 GB	100%	`google/gemma-4-31B-it`
FP8-dynamic	33.3 GB	53%	RedHatAI/gemma-4-31B-it-FP8-Dynamic — 89.6% FP8_E4M3 + 6.1% BF16
This Model (MXFP4)	19.5 GB	31%	W4A16 GPTQ + MHR + PSA

Citation

@misc{gemma4-mxfp4-mhr-psa,
  title={Gemma 4 31B IT MXFP4 with Multilingual Hessian Regularization and PSA-GPTQ},
  author={Taeyoung Lee},
  year={2026},
  url={https://huggingface.co/BCCard/MoAI-gemma-4-31B-it-mxfp4}
}