mrysh/gemma-4-e4b-it-nvfp4

gemma-4-e4b-it - nvfp4

Quantized version of google/gemma-4-E4B-it made for a quantization benchmark (4 models x 8 techniques, evaluated with vLLM on BFCL, Energy-QA, hiring-bias and MedQA).

• method: nvfp4, about 4 bits, group size 16
• library: nvidia-modelopt 0.47.0
• calibration samples: 512
• size on disk: 9.76 GB

NVFP4 (NVIDIA 4-bit floating point, group size 16, made with NVIDIA ModelOpt, calibration set of 512 mixed samples). Loads in vLLM (vllm serve <repo>). The speed gain needs a Blackwell GPU (e.g. RTX PRO 6000, B200). The vision/audio parts are kept in 16 bit.

Evaluated as a text model (vLLM, thinking off). Same license as the base model; see the base model card.