medgemma-auto-fp8

google/medgemma-27b-text-it quantized to FP8 (W8A8 dynamic) with llm-compressor, in compressed-tensors format.

Why this exists

Serving the BF16 checkpoint with --quantization fp8 on a 48 GB card OOMs: SGLang/vLLM allocate params_dtype (BF16, 51 GiB for 27B) in create_weights and only compress after loading. A pre-quantized checkpoint makes create_weights allocate FP8 directly (26 GiB). Measured peak while producing this artifact: 51.0 GiB VRAM.

Recipe

QuantizationModifier(targets="Linear", scheme="FP8_DYNAMIC", ignore=['lm_head', 're:.*embed_tokens.*', 're:model\\.layers\\.0\\..*'])

• weights: 8-bit float, per-channel, symmetric
• activations: 8-bit float, dynamic per-token (data-free, no calibration set)
• left in BF16: lm_head (tied to embed_tokens in Gemma 3), embed_tokens, and decoder layer 0 (the most quantization-sensitive block)

Serving with SGLang

python -m sglang.launch_server --model-path fhai50032/medgemma-auto-fp8 --kv-cache-dtype fp8_e4m3 --attention-backend triton --context-length 65536

Do not pass --quantization fp8 - it is auto-detected from quantization_config, and passing it explicitly can select the wrong quant method for a compressed-tensors checkpoint.

Use --attention-backend triton: FlashInfer silently disables Gemma 3 sliding window (vLLM #20865). SGLang also disables hybrid SWA memory for all Gemma 2/3/3n architectures, so every layer caches the full context - budget KV cache accordingly.

Accuracy

Not formally evaluated. No recovery figure is claimed for this artifact.