Add MLX quantized model with KV cache compression

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,105 @@
+---
+base_model: google/gemma-4-26B-A4B
+library_name: mlx
+tags:
+  - rotorquant
+  - kv-cache-quantization
+  - gemma
+  - gemma4
+  - multimodal
+  - quantized
+  - mlx
+  - 2bit
+license: apache-2.0
+pipeline_tag: image-text-to-text
+---
+
+# Gemma 4 26B-A4B - RotorQuant MLX 2-bit
+
+**2-bit weight-quantized MLX version** of [google/gemma-4-26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B) with RotorQuant KV-cache quantization. Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. The most aggressive quantization, fitting the full model in the smallest possible footprint. Only 4B parameters are active per token despite 26B total, making this model significantly more efficient at inference time than its parameter count suggests.
+
+Approximate model size: **~7 GB**
+
+## Model Specifications
+
+| Property | Value |
+|---|---|
+| **Base Model** | [google/gemma-4-26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B) |
+| **Parameters** | 26 billion total (4 billion active per token) |
+| **Architecture** | Mixture-of-Experts (MoE) (4B active per token) |
+| **Modality** | Multimodal: image + text input, text output |
+| **License** | Apache 2.0 |
+| **Weight Quantization** | 2-bit (~7 GB) |
+| **KV-Cache Quantization** | RotorQuant |
+| **Framework** | MLX (Apple Silicon) |
+
+## Quickstart
+
+```python
+import mlx.core as mx
+from mlx_lm import load, generate
+
+model, tokenizer = load("majentik/gemma-4-26B-A4B-RotorQuant-MLX-2bit")
+
+prompt = "The history of artificial intelligence began"
+response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
+print(response)
+```
+
+For multimodal usage with images:
+
+```python
+from mlx_vlm import load, generate
+
+model, processor = load("majentik/gemma-4-26B-A4B-RotorQuant-MLX-2bit")
+
+prompt = "Describe the contents of this image."
+output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", max_tokens=512)
+print(output)
+```
+
+## What is RotorQuant?
+
+[RotorQuant](https://github.com/scrya-com/rotorquant) is a high-performance KV-cache quantization method that achieves significantly better throughput than TurboQuant. Combined with 2-bit weight quantization in MLX, this provides maximum compression with the best available KV-cache performance: the smallest possible model footprint plus the fastest compressed KV cache for efficient long-context generation.
+
+Key advantages over TurboQuant:
+- **5.3x faster prefill**
+- **28% faster decode**
+- Equivalent memory savings
+
+**Note:** 2-bit quantization is the most aggressive option and may result in some quality degradation compared to higher-precision variants. It is best suited for experimentation, rapid prototyping, or hardware-constrained environments.
+
+## KV-Cache Quantization Comparison
+
+| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
+|---|---|---|---|---|
+| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
+| **RotorQuant** | **5.3x faster** | **28% faster** | High | [GitHub](https://github.com/scrya-com/rotorquant) |
+
+## Memory Estimates (Gemma 4 26B-A4B)
+
+| Precision | Approximate Size | MLX Variant |
+|---|---|---|
+| FP16 (original) | ~52 GB | -- |
+| 8-bit quantized | ~26 GB | [RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-26B-A4B-RotorQuant-MLX-8bit) |
+| 4-bit quantized | ~14 GB | [RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-26B-A4B-RotorQuant-MLX-4bit) |
+| **2-bit quantized** | **~7 GB** | **This model** |
+
+## Hardware Requirements
+
+This model requires approximately 7 GB of unified memory. Recommended hardware:
+- Apple M1 (16 GB+)
+- Apple M2 (16 GB+)
+- Apple M3 (16 GB+)
+- Apple M4 (16 GB+)
+- Any Apple Silicon Mac with 16 GB+ unified memory
+
+## See Also
+
+- [google/gemma-4-26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B) -- Base model
+- [majentik/gemma-4-26B-A4B-RotorQuant](https://huggingface.co/majentik/gemma-4-26B-A4B-RotorQuant) -- RotorQuant KV-cache only (transformers)
+- [majentik/gemma-4-26B-A4B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-26B-A4B-RotorQuant-MLX-8bit) -- MLX 8-bit variant
+- [majentik/gemma-4-26B-A4B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-26B-A4B-RotorQuant-MLX-4bit) -- MLX 4-bit variant
+- [majentik/gemma-4-26B-A4B-TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-26B-A4B-TurboQuant-MLX-2bit) -- TurboQuant MLX 2-bit variant
+- [RotorQuant GitHub](https://github.com/scrya-com/rotorquant)
+- [MLX Framework](https://github.com/ml-explore/mlx)

config.json

@@ -0,0 +1,1113 @@
+{
+  "architectures": [
+    "Gemma4ForConditionalGeneration"
+  ],
+  "audio_config": null,
+  "audio_token_id": 258881,
+  "boa_token_id": 256000,
+  "boi_token_id": 255999,
+  "dtype": "bfloat16",
+  "eoa_token_id": 258883,
+  "eoa_token_index": 258883,
+  "eoi_token_id": 258882,
+  "eos_token_id": 1,
+  "image_token_id": 258880,
+  "initializer_range": 0.02,
+  "model_type": "gemma4",
+  "quantization": {
+    "group_size": 64,
+    "bits": 4,
+    "mode": "affine",
+    "language_model.model.layers.0.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    }
+  },
+  "quantization_config": {
+    "group_size": 64,
+    "bits": 2,
+    "mode": "affine",
+    "language_model.model.layers.0.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.0.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.1.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.2.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.3.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.4.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.5.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.6.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.7.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.8.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.9.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.10.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.11.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.12.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.13.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.14.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.15.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.16.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.17.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.18.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.19.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.20.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.21.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.22.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.23.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.24.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.25.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.26.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.27.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.28.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.gate_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.down_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.mlp.up_proj": {
+      "group_size": 64,
+      "bits": 8
+    },
+    "language_model.model.layers.29.router.proj": {
+      "group_size": 64,
+      "bits": 8
+    }
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attention_k_eq_v": true,
+    "bos_token_id": 2,
+    "dtype": "bfloat16",
+    "enable_moe_block": true,
+    "eos_token_id": 1,
+    "final_logit_softcapping": 30.0,
+    "global_head_dim": 512,
+    "head_dim": 256,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 2816,
+    "hidden_size_per_layer_input": 0,
+    "initializer_range": 0.02,
+    "intermediate_size": 2112,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 262144,
+    "model_type": "gemma4_text",
+    "moe_intermediate_size": 704,
+    "num_attention_heads": 16,
+    "num_experts": 128,
+    "num_global_key_value_heads": 2,
+    "num_hidden_layers": 30,
+    "num_key_value_heads": 8,
+    "num_kv_shared_layers": 0,
+    "pad_token_id": 0,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "full_attention": {
+        "partial_rotary_factor": 0.25,
+        "rope_theta": 1000000.0,
+        "rope_type": "proportional"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000.0,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 1024,
+    "tie_word_embeddings": true,
+    "top_k_experts": 8,
+    "use_bidirectional_attention": "vision",
+    "use_cache": true,
+    "use_double_wide_mlp": false,
+    "vocab_size": 262144,
+    "vocab_size_per_layer_input": 262144
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.5.0.dev0",
+  "video_token_id": 258884,
+  "vision_config": {
+    "_name_or_path": "",
+    "architectures": null,
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "chunk_size_feed_forward": 0,
+    "default_output_length": 280,
+    "dtype": "bfloat16",
+    "global_head_dim": 72,
+    "head_dim": 72,
+    "hidden_activation": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "max_position_embeddings": 131072,
+    "model_type": "gemma4_vision",
+    "num_attention_heads": 16,
+    "num_hidden_layers": 27,
+    "num_key_value_heads": 16,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "position_embedding_size": 10240,
+    "problem_type": null,
+    "return_dict": true,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "rope_theta": 100.0,
+      "rope_type": "default"
+    },
+    "standardize": true,
+    "use_clipped_linears": false
+  },
+  "vision_soft_tokens_per_image": 280
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "bos_token_id": 2,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "temperature": 1.0,
+  "top_k": 64,
+  "top_p": 0.95,
+  "transformers_version": "5.5.0.dev0"
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68015cddbe0df67a9449e3cae3e1678ca4b40b02df5624d3bee2a10fa61e27af
+size 2938095734

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c23438bccc430e25c989e24a30bf6e163bb6925750e8abc89ef1a2d4a3761bf
+size 2949829866

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71859baf778f361c039751d289382e68209f39dddcca1e9206f95a248921975f
+size 3309601180

processor_config.json

@@ -0,0 +1,32 @@
+{
+  "audio_seq_length": 750,
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": false,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.0,
+      0.0,
+      0.0
+    ],
+    "image_processor_type": "Gemma4ImageProcessor",
+    "image_seq_length": 280,
+    "image_std": [
+      1.0,
+      1.0,
+      1.0
+    ],
+    "max_soft_tokens": 280,
+    "patch_size": 16,
+    "pooling_kernel_size": 3,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "height": 224,
+      "width": 224
+    }
+  },
+  "image_seq_length": 280,
+  "processor_class": "Gemma4Processor"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12bac982b793c44b03d52a250a9f0d0b666813da566b910c24a6da0695fd11e6
+size 32170070

tokenizer_config.json

@@ -0,0 +1,54 @@
+{
+  "audio_token": "<|audio|>",
+  "backend": "tokenizers",
+  "boa_token": "<|audio>",
+  "boi_token": "<|image>",
+  "bos_token": "<bos>",
+  "eoa_token": "<audio|>",
+  "eoc_token": "<channel|>",
+  "eoi_token": "<image|>",
+  "eos_token": "<eos>",
+  "eot_token": "<turn|>",
+  "escape_token": "<|\"|>",
+  "etc_token": "<tool_call|>",
+  "etd_token": "<tool|>",
+  "etr_token": "<tool_response|>",
+  "extra_special_tokens": [
+    "<|video|>"
+  ],
+  "image_token": "<|image|>",
+  "is_local": true,
+  "mask_token": "<mask>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "audio_token": "<|audio|>",
+    "boa_token": "<|audio>",
+    "boi_token": "<|image>",
+    "eoa_token": "<audio|>",
+    "eoc_token": "<channel|>",
+    "eoi_token": "<image|>",
+    "eot_token": "<turn|>",
+    "escape_token": "<|\"|>",
+    "etc_token": "<tool_call|>",
+    "etd_token": "<tool|>",
+    "etr_token": "<tool_response|>",
+    "image_token": "<|image|>",
+    "soc_token": "<|channel>",
+    "sot_token": "<|turn>",
+    "stc_token": "<|tool_call>",
+    "std_token": "<|tool>",
+    "str_token": "<|tool_response>",
+    "think_token": "<|think|>"
+  },
+  "pad_token": "<pad>",
+  "padding_side": "left",
+  "processor_class": "Gemma4Processor",
+  "soc_token": "<|channel>",
+  "sot_token": "<|turn>",
+  "stc_token": "<|tool_call>",
+  "std_token": "<|tool>",
+  "str_token": "<|tool_response>",
+  "think_token": "<|think|>",
+  "tokenizer_class": "GemmaTokenizer",
+  "unk_token": "<unk>"
+}