File size: 3,204 Bytes
f897353 00764dc f897353 00764dc 2e83567 0771c47 ab8b0a7 0771c47 21509e0 0771c47 21509e0 0771c47 ab8b0a7 0771c47 21509e0 4239280 f897353 00764dc f897353 21509e0 f897353 0771c47 21509e0 2bac654 1e15ed1 0771c47 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
license: apache-2.0
base_model: google/gemma-4-E2B-it
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- gemma4
- monarch-matrices
- model-compression
---
# Gemma 4 E2B Distilled
Distilled Gemma 4 E2B with all 35 language-model MLPs replaced by two-factor
Monarch maps. The released BF16 model has `3,682,268,704` parameters, instead
of `5,104,297,504` in the original Gemma 4 E2B.
## 1. Weight Storage
| Model | Status | Loaded weights | Serialized weights | Reduction vs dense BF16 |
| --- | --- | ---: | ---: | ---: |
| Dense Gemma 4 BF16 | Reference | 9.507 GiB | 9.543 GiB | - |
| Distilled Gemma 4 BF16 | Released | 6.859 GiB | 6.859 GiB | 27.86% |
| Distilled Gemma 4 + LoRA r8 BF16 | Experimental | 6.876 GiB | 6.877 GiB | 27.68% |
| Distilled Gemma 4 + INT8 linears | Released | 6.135 GiB | 6.136 GiB | 35.48% |
BF16 sizes use two bytes per parameter; serialized LoRA size was audited from
the two safetensor shards. Values exclude activations and CUDA workspaces.
## 2. TinyHellaSwag
All rows used the same RTX PRO 6000, fixed batch size `32`, seed `1234`, 100
official examples, 10-shot prompts, and no chat template.
| Model | Status | GP-IRT accuracy | Raw accuracy | Runtime | Peak VRAM |
| --- | --- | ---: | ---: | ---: | ---: |
| Dense Gemma 4 BF16 | Reference | 39.23% | 29% | 28.50 s | 57.88 GiB |
| Distilled Gemma 4 BF16 | Released | 32.35% | 22% | 23.33 s | 55.23 GiB |
| Distilled Gemma 4 + LoRA r8 BF16 | Experimental | 33.25% | 23% | 24.93 s | 55.25 GiB |
| Distilled Gemma 4 + INT8 linears | Released | 30.57% | 21% | 23.14 s | 54.51 GiB |
Rank-8 LoRA improved the 35-layer source by `0.90` GP-IRT points and one correct
item, but missed the former `+1.0` point or `+2` item release gate. The
100-example benchmark is noisy.
## Usage
```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "hexoy/gemma-4-e2b-distilled"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
trust_remote_code=True,
dtype="auto",
device_map="auto",
)
```
The rank-8 result is published separately at
[`hexoy/gemma-4-e2b-monarch-35mlp-lora-r8`](https://huggingface.co/hexoy/gemma-4-e2b-monarch-35mlp-lora-r8).
It is a benchmark-recovery experiment: corrected 64-token generation repeated
phrases for text and remained malformed or inaccurate for images. It is not a
general text or multimodal recovery release.
## Reproducibility
- BF16 model revision: `f897353fca328b1cc5fd2e12d645773ca637f5f0`
- GitHub repository: [`ratmir-miftachov/gemma-distillation`](https://github.com/ratmir-miftachov/gemma-distillation)
- INT8 variant: [`hexoy/gemma-4-e2b-monarch-35mlp-int8`](https://huggingface.co/hexoy/gemma-4-e2b-monarch-35mlp-int8)
- Experimental LoRA r8 variant: [`hexoy/gemma-4-e2b-monarch-35mlp-lora-r8`](https://huggingface.co/hexoy/gemma-4-e2b-monarch-35mlp-lora-r8)
- LoRA source commit: `1435571b20dd26c073a535678975884154add5b8`
Detailed training and release evidence is retained privately.
Derived from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it).
See `NOTICE` for the modification summary.
|