Instructions to use hexoy/gemma-4-e2b-monarch-35mlp-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hexoy/gemma-4-e2b-monarch-35mlp-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hexoy/gemma-4-e2b-monarch-35mlp-int8", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("hexoy/gemma-4-e2b-monarch-35mlp-int8", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hexoy/gemma-4-e2b-monarch-35mlp-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hexoy/gemma-4-e2b-monarch-35mlp-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-monarch-35mlp-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hexoy/gemma-4-e2b-monarch-35mlp-int8

SGLang

How to use hexoy/gemma-4-e2b-monarch-35mlp-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hexoy/gemma-4-e2b-monarch-35mlp-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-monarch-35mlp-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hexoy/gemma-4-e2b-monarch-35mlp-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-monarch-35mlp-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hexoy/gemma-4-e2b-monarch-35mlp-int8 with Docker Model Runner:
```
docker model run hf.co/hexoy/gemma-4-e2b-monarch-35mlp-int8
```

Configuration Parsing Warning:In config.json: "quantization_config.modules_to_not_convert" must be an array

Gemma 4 E2B Distilled INT8

Experimental weight-only INT8 version of hexoy/gemma-4-e2b-distilled at immutable source revision f897353fca328b1cc5fd2e12d645773ca637f5f0.

All 420 supported, untied standard nn.Linear weights across language, vision, audio, and multimodal components use TorchAO 0.17.0 symmetric per-output-channel INT8 weight-only quantization. All 210 trained Monarch factor tensors, the tied embedding/LM-head weight, normalization parameters, biases, unsupported non-linear parameters, and runtime activations remain BF16.

Architecture And Weight Storage

Model stage	Parameters	Monarch MLPs	INT8 linears	PLE storage	Loaded weights	Serialized weights	Loaded reduction vs dense
Dense Gemma 4 BF16	5,104,297,504	0	0	BF16	9.507 GiB	9.543 GiB	-
Distilled Gemma 4 BF16	3,682,268,704	35	0	BF16	6.859 GiB	6.859 GiB	27.86%
Distilled Gemma 4 + INT8 linears	3,682,268,704	35	420	BF16	6.135 GiB	6.136 GiB	35.48%

The BF16 loaded footprints are parameter storage at two bytes per parameter. The quantized footprint was audited from physical TorchAO tensors. Serialized values count safetensor files only. None of these values includes activations, logits, temporary buffers, CUDA workspaces, or allocator overhead.

The INT8 model contains 779,419,648 quantized linear weights. The BF16 remainder contains 2,902,849,056 parameters (5.41 GiB), including 135,106,560 Monarch factor parameters. Quantized physical storage is measured from the TorchAO qdata, scale, zero-point, BF16 parameter, and buffer tensors; it is not the logical BF16 shape reported by generic PyTorch tensor-subclass accounting.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "hexoy/gemma-4-e2b-monarch-35mlp-int8"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
)

This export requires torchao==0.17.0 and contains custom modeling code, so review it before enabling trust_remote_code=True. INT8 applies to weights only; activations and logits remain BF16.

Related models: BF16 Distilled Gemma 4 and experimental LoRA r8.

TinyHellaSwag Benchmark

All rows used the same official tinyHellaswag protocol: 100 anchor examples, 10-shot prompts, character-length-normalized continuation likelihood, no chat template, seed 1234, BF16 activations, fixed batch size 32, and an NVIDIA RTX PRO 6000 Blackwell GPU.

Model	GP-IRT estimated accuracy	Raw anchor accuracy	Runtime	Peak GPU memory	Batch
`google/gemma-4-E2B-it`	39.23%	29%	28.50 s	57.88 GiB	32
`hexoy/gemma-4-e2b-distilled`	32.35%	22%	23.33 s	55.23 GiB	32
`hexoy/gemma-4-e2b-monarch-35mlp-int8`	30.57%	21%	23.14 s	54.51 GiB	32

Relative to BF16 Distilled Gemma 4, INT8 changed GP-IRT by -1.78 percentage points and raw accuracy by -1 point. The paired bootstrap 95% interval was [-3, 0] points, with two disagreements and McNemar p=1.0. Relative to dense, the INT8 deltas were -8.66 and -8 points ([-18, +2], McNemar p=0.1849).

Pinned model revisions:

Original: 9dbdf8a839e4e9e0eb56ed80cc8886661d3817cf
BF16 35-layer: f897353fca328b1cc5fd2e12d645773ca637f5f0
INT8 weights: db56825e2e0de59115049d7109632b2f1ce80905

Complete results, paired comparisons, logs, environment metadata, inventory, and checksums are retained in a private artifact repository.

Prompted MNLI Benchmark

Prompted MNLI used all 9,815 examples from the pinned GLUE MNLI validation_matched split, mean continuation-token likelihood, a 128-token context, and no chat template.

Model	Accuracy	Macro F1	Runtime	Peak GPU memory	Batch
`google/gemma-4-E2B-it`	34.80%	32.83%	66.5 s	17.56 GiB	64
`hexoy/gemma-4-e2b-distilled`	35.36%	18.13%	67.3 s	14.91 GiB	64
`hexoy/gemma-4-e2b-monarch-35mlp-int8`	Not run	Not run	-	-	-

The INT8 model was not evaluated on prompted MNLI. The 35-layer BF16 result exhibits severe class collapse: it predicted entailment for 9,701 of 9,815 examples, so macro F1 is more informative than its raw accuracy.

Limitations

This is an experimental compression and quantization artifact, not an official Google model.
TinyHellaSwag has only 100 anchor items; small accuracy changes have wide uncertainty.
Weight-only INT8 reduces persistent weights, while activation-heavy peak VRAM falls by less.
Evaluate the model on the intended task before deployment.

Attribution

Derived from google/gemma-4-E2B-it through the BF16 Monarch model. See NOTICE for the modification summary.

Downloads last month: 61

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for hexoy/gemma-4-e2b-monarch-35mlp-int8

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Finetuned

hexoy/gemma-4-e2b-distilled

Quantized

(1)

this model