Instructions to use hexoy/gemma-4-e2b-distilled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hexoy/gemma-4-e2b-distilled with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="hexoy/gemma-4-e2b-distilled", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained("hexoy/gemma-4-e2b-distilled", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use hexoy/gemma-4-e2b-distilled with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hexoy/gemma-4-e2b-distilled"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-distilled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/hexoy/gemma-4-e2b-distilled

SGLang

How to use hexoy/gemma-4-e2b-distilled with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hexoy/gemma-4-e2b-distilled" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-distilled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hexoy/gemma-4-e2b-distilled" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hexoy/gemma-4-e2b-distilled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use hexoy/gemma-4-e2b-distilled with Docker Model Runner:
```
docker model run hf.co/hexoy/gemma-4-e2b-distilled
```

gemma-4-e2b-distilled / README.md

Commit History

Clarify GitHub repository link

2bac654
verified

hexoy commited on 11 days ago

Add original parameter comparison

2e83567
verified

hexoy commited on 14 days ago

Rename base model to Distilled Gemma 4

00764dc
verified

hexoy commited on 14 days ago

Clarify distilled model labels

ab8b0a7
verified

hexoy commited on 14 days ago

Remove four-layer comparison results

eef9042
verified

hexoy commited on 18 days ago

Prepare model card for public release

1e15ed1
verified

hexoy commited on 19 days ago

Add experimental rank-8 LoRA comparison

21509e0
verified

hexoy commited on 19 days ago

Remove rejected PLE experiment references

a63d660
verified

hexoy commited on 20 days ago

Document rejected group-32 INT8 PLE trial

8f6ba52
verified

hexoy commited on 20 days ago

Remove Unsloth results from model card

555ca03
verified

hexoy commited on 20 days ago

Add Unsloth storage and TinyHellaSwag results

1dba879
verified

hexoy commited on 21 days ago

Simplify storage and TinyHellaSwag results

0771c47
verified

hexoy commited on 21 days ago

Document controlled PLE v2 and GGUF experiments

7181d2f
verified

hexoy commited on 21 days ago

Synchronize quantization comparison tables

db2cb85
verified

hexoy commited on 21 days ago

Add experimental PLE INT8 benchmark result

a9a2931
verified

hexoy commited on 21 days ago

Add INT8 storage and TinyHellaSwag comparison

91e4ba4
verified

hexoy commited on 23 days ago

Document parameter and BF16 weight memory reduction

4239280
verified

hexoy commited on 23 days ago

Document TinyHellaSwag and MNLI benchmarks

1e1954a

Ratmir Miftachov commited on 24 days ago

Document completed all-35 distillation run

bbdd8c5
verified

hexoy commited on 25 days ago

Upload folder using huggingface_hub

f897353
verified

hexoy commited on 25 days ago

Commit History

Clarify GitHub repository link 2bac654 verified

Add original parameter comparison 2e83567 verified

Rename base model to Distilled Gemma 4 00764dc verified

Clarify distilled model labels ab8b0a7 verified

Remove four-layer comparison results eef9042 verified

Prepare model card for public release 1e15ed1 verified

Add experimental rank-8 LoRA comparison 21509e0 verified

Remove rejected PLE experiment references a63d660 verified

Document rejected group-32 INT8 PLE trial 8f6ba52 verified

Remove Unsloth results from model card 555ca03 verified

Add Unsloth storage and TinyHellaSwag results 1dba879 verified

Simplify storage and TinyHellaSwag results 0771c47 verified

Document controlled PLE v2 and GGUF experiments 7181d2f verified

Synchronize quantization comparison tables db2cb85 verified

Add experimental PLE INT8 benchmark result a9a2931 verified

Add INT8 storage and TinyHellaSwag comparison 91e4ba4 verified

Document parameter and BF16 weight memory reduction 4239280 verified

Document TinyHellaSwag and MNLI benchmarks 1e1954a

Document completed all-35 distillation run bbdd8c5 verified

Upload folder using huggingface_hub f897353 verified

Clarify GitHub repository link

2bac654
verified

Add original parameter comparison

2e83567
verified

Rename base model to Distilled Gemma 4

00764dc
verified

Clarify distilled model labels

ab8b0a7
verified

Remove four-layer comparison results

eef9042
verified

Prepare model card for public release

1e15ed1
verified

Add experimental rank-8 LoRA comparison

21509e0
verified

Remove rejected PLE experiment references

a63d660
verified

Document rejected group-32 INT8 PLE trial

8f6ba52
verified

Remove Unsloth results from model card

555ca03
verified

Add Unsloth storage and TinyHellaSwag results

1dba879
verified

Simplify storage and TinyHellaSwag results

0771c47
verified

Document controlled PLE v2 and GGUF experiments

7181d2f
verified

Synchronize quantization comparison tables

db2cb85
verified

Add experimental PLE INT8 benchmark result

a9a2931
verified

Add INT8 storage and TinyHellaSwag comparison

91e4ba4
verified

Document parameter and BF16 weight memory reduction

4239280
verified

Document TinyHellaSwag and MNLI benchmarks

1e1954a

Document completed all-35 distillation run

bbdd8c5
verified

Upload folder using huggingface_hub

f897353
verified