Image-Text-to-Text
MLX
Safetensors
gemma4
rotorquant
kv-cache-quantization
gemma
multimodal
quantized
2bit
2-bit
Instructions to use majentik/gemma-4-E4B-RotorQuant-MLX-2bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use majentik/gemma-4-E4B-RotorQuant-MLX-2bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("majentik/gemma-4-E4B-RotorQuant-MLX-2bit") config = load_config("majentik/gemma-4-E4B-RotorQuant-MLX-2bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Card accuracy sweep: honest brand labeling, remove dead links, upstream KV tip
Browse files
README.md
CHANGED
|
@@ -33,7 +33,7 @@ pipeline_tag: image-text-to-text
|
|
| 33 |
|
| 34 |
# Gemma 4 E4B - RotorQuant MLX 2-bit
|
| 35 |
|
| 36 |
-
**2-bit weight-quantized MLX version** of [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) with RotorQuant KV-cache
|
| 37 |
|
| 38 |
Approximate model size: **~1.2 GB**
|
| 39 |
|
|
@@ -75,23 +75,21 @@ output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", ma
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
-
##
|
| 79 |
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
-
|
| 84 |
-
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
**Note:** 2-bit quantization is the most aggressive option and may result in some quality degradation compared to higher-precision variants. It is best suited for experimentation, rapid prototyping, or hardware-constrained environments.
|
| 88 |
|
| 89 |
## KV-Cache Quantization Comparison
|
| 90 |
|
| 91 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 92 |
|---|---|---|---|---|
|
| 93 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 94 |
-
|
| 95 |
|
| 96 |
## Memory Estimates (Gemma 4 E4B)
|
| 97 |
|
|
@@ -114,7 +112,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
|
|
| 114 |
## See Also
|
| 115 |
|
| 116 |
- [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) -- Base model
|
| 117 |
-
- [majentik/gemma-4-E4B-RotorQuant](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant) -- RotorQuant KV-cache only (transformers)
|
| 118 |
- [majentik/gemma-4-E4B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-8bit) -- MLX 8-bit variant
|
| 119 |
- [majentik/gemma-4-E4B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-4bit) -- MLX 4-bit variant
|
| 120 |
- [majentik/gemma-4-E4B-TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-E4B-TurboQuant-MLX-2bit) -- TurboQuant MLX 2-bit variant
|
|
@@ -140,7 +137,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
|
|
| 140 |
|
| 141 |
| Variant | Runtime | Approx size | Use case |
|
| 142 |
|---|---|---|---|
|
| 143 |
-
| [RotorQuant](https://huggingface.co/majentik/gemma-4-e4b-rotorquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 144 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~3.4 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 145 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q2_K) | llama.cpp | ~2.4 GB | Lossy, low-RAM CPU/edge |
|
| 146 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~3.1 GB | Smaller 3-bit, CPU-friendly |
|
|
@@ -153,4 +149,4 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
|
|
| 153 |
| [TurboQuant](https://huggingface.co/majentik/gemma-4-e4b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 154 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-2bit) | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
|
| 155 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-4bit) | mlx-lm | ~2.5 GB | Apple Silicon balanced |
|
| 156 |
-
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-8bit) | mlx-lm | ~4.7 GB | Apple Silicon reference |
|
|
|
|
| 33 |
|
| 34 |
# Gemma 4 E4B - RotorQuant MLX 2-bit
|
| 35 |
|
| 36 |
+
**2-bit weight-quantized MLX version** of [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. The most aggressive quantization, fitting the full model in the smallest possible footprint.
|
| 37 |
|
| 38 |
Approximate model size: **~1.2 GB**
|
| 39 |
|
|
|
|
| 75 |
print(output)
|
| 76 |
```
|
| 77 |
|
| 78 |
+
## About the RotorQuant / TurboQuant labels
|
| 79 |
|
| 80 |
+
RotorQuant and TurboQuant are this project's **release labels**, not distinct
|
| 81 |
+
quantization algorithms — for any given tier, both brand repos carry
|
| 82 |
+
byte-identical weights produced with the standard MLX / llama.cpp quantizers.
|
| 83 |
+
No brand-specific speedup is claimed or measured. The KV-cache fork these
|
| 84 |
+
labels originally referred to is legacy; for KV-cache memory savings use the
|
| 85 |
+
upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).
|
|
|
|
|
|
|
| 86 |
|
| 87 |
## KV-Cache Quantization Comparison
|
| 88 |
|
| 89 |
| Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
|
| 90 |
|---|---|---|---|---|
|
| 91 |
| **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
|
| 92 |
+
|
| 93 |
|
| 94 |
## Memory Estimates (Gemma 4 E4B)
|
| 95 |
|
|
|
|
| 112 |
## See Also
|
| 113 |
|
| 114 |
- [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) -- Base model
|
|
|
|
| 115 |
- [majentik/gemma-4-E4B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-8bit) -- MLX 8-bit variant
|
| 116 |
- [majentik/gemma-4-E4B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-4bit) -- MLX 4-bit variant
|
| 117 |
- [majentik/gemma-4-E4B-TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-E4B-TurboQuant-MLX-2bit) -- TurboQuant MLX 2-bit variant
|
|
|
|
| 137 |
|
| 138 |
| Variant | Runtime | Approx size | Use case |
|
| 139 |
|---|---|---|---|
|
|
|
|
| 140 |
| [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~3.4 GB | Lossy 4-bit, low-RAM CPU/edge |
|
| 141 |
| [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q2_K) | llama.cpp | ~2.4 GB | Lossy, low-RAM CPU/edge |
|
| 142 |
| [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~3.1 GB | Smaller 3-bit, CPU-friendly |
|
|
|
|
| 149 |
| [TurboQuant](https://huggingface.co/majentik/gemma-4-e4b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
|
| 150 |
| [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-2bit) | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
|
| 151 |
| [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-4bit) | mlx-lm | ~2.5 GB | Apple Silicon balanced |
|
| 152 |
+
| [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-8bit) | mlx-lm | ~4.7 GB | Apple Silicon reference |
|