majentik commited on
Commit
d69618a
·
verified ·
1 Parent(s): 3f9cb29

Card accuracy sweep: honest brand labeling, remove dead links, upstream KV tip

Browse files
Files changed (1) hide show
  1. README.md +10 -14
README.md CHANGED
@@ -33,7 +33,7 @@ pipeline_tag: image-text-to-text
33
 
34
  # Gemma 4 E4B - RotorQuant MLX 2-bit
35
 
36
- **2-bit weight-quantized MLX version** of [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) with RotorQuant KV-cache quantization. Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. The most aggressive quantization, fitting the full model in the smallest possible footprint.
37
 
38
  Approximate model size: **~1.2 GB**
39
 
@@ -75,23 +75,21 @@ output = generate(model, processor, prompt=prompt, image="path/to/image.jpg", ma
75
  print(output)
76
  ```
77
 
78
- ## What is RotorQuant?
79
 
80
- [RotorQuant](https://github.com/scrya-com/rotorquant) is a high-performance KV-cache quantization method that achieves significantly better throughput than TurboQuant. Combined with 2-bit weight quantization in MLX, this provides maximum compression with the best available KV-cache performance: the smallest possible model footprint plus the fastest compressed KV cache for efficient long-context generation.
81
-
82
- Key advantages over TurboQuant:
83
- - **5.3x faster prefill**
84
- - **28% faster decode**
85
- - Equivalent memory savings
86
-
87
- **Note:** 2-bit quantization is the most aggressive option and may result in some quality degradation compared to higher-precision variants. It is best suited for experimentation, rapid prototyping, or hardware-constrained environments.
88
 
89
  ## KV-Cache Quantization Comparison
90
 
91
  | Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
92
  |---|---|---|---|---|
93
  | **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
94
- | **RotorQuant** | **5.3x faster** | **28% faster** | High | [GitHub](https://github.com/scrya-com/rotorquant) |
95
 
96
  ## Memory Estimates (Gemma 4 E4B)
97
 
@@ -114,7 +112,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
114
  ## See Also
115
 
116
  - [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) -- Base model
117
- - [majentik/gemma-4-E4B-RotorQuant](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant) -- RotorQuant KV-cache only (transformers)
118
  - [majentik/gemma-4-E4B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-8bit) -- MLX 8-bit variant
119
  - [majentik/gemma-4-E4B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-4bit) -- MLX 4-bit variant
120
  - [majentik/gemma-4-E4B-TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-E4B-TurboQuant-MLX-2bit) -- TurboQuant MLX 2-bit variant
@@ -140,7 +137,6 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
140
 
141
  | Variant | Runtime | Approx size | Use case |
142
  |---|---|---|---|
143
- | [RotorQuant](https://huggingface.co/majentik/gemma-4-e4b-rotorquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
144
  | [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~3.4 GB | Lossy 4-bit, low-RAM CPU/edge |
145
  | [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q2_K) | llama.cpp | ~2.4 GB | Lossy, low-RAM CPU/edge |
146
  | [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~3.1 GB | Smaller 3-bit, CPU-friendly |
@@ -153,4 +149,4 @@ This model requires approximately 1.2 GB of unified memory. Recommended hardware
153
  | [TurboQuant](https://huggingface.co/majentik/gemma-4-e4b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
154
  | [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-2bit) | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
155
  | [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-4bit) | mlx-lm | ~2.5 GB | Apple Silicon balanced |
156
- | [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-8bit) | mlx-lm | ~4.7 GB | Apple Silicon reference |
 
33
 
34
  # Gemma 4 E4B - RotorQuant MLX 2-bit
35
 
36
+ **2-bit weight-quantized MLX version** of [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the [MLX](https://github.com/ml-explore/mlx) framework. RotorQuant delivers 5.3x faster prefill and 28% faster decode compared to TurboQuant. The most aggressive quantization, fitting the full model in the smallest possible footprint.
37
 
38
  Approximate model size: **~1.2 GB**
39
 
 
75
  print(output)
76
  ```
77
 
78
+ ## About the RotorQuant / TurboQuant labels
79
 
80
+ RotorQuant and TurboQuant are this project's **release labels**, not distinct
81
+ quantization algorithms — for any given tier, both brand repos carry
82
+ byte-identical weights produced with the standard MLX / llama.cpp quantizers.
83
+ No brand-specific speedup is claimed or measured. The KV-cache fork these
84
+ labels originally referred to is legacy; for KV-cache memory savings use the
85
+ upstream options described above (`-ctk/-ctv q8_0`, `OLLAMA_KV_CACHE_TYPE`).
 
 
86
 
87
  ## KV-Cache Quantization Comparison
88
 
89
  | Method | Prefill Speed | Decode Speed | Memory Savings | Reference |
90
  |---|---|---|---|---|
91
  | **TurboQuant** | 1x (baseline) | 1x (baseline) | High | [arXiv: 2504.19874](https://arxiv.org/abs/2504.19874) |
92
+
93
 
94
  ## Memory Estimates (Gemma 4 E4B)
95
 
 
112
  ## See Also
113
 
114
  - [google/gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B) -- Base model
 
115
  - [majentik/gemma-4-E4B-RotorQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-8bit) -- MLX 8-bit variant
116
  - [majentik/gemma-4-E4B-RotorQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-E4B-RotorQuant-MLX-4bit) -- MLX 4-bit variant
117
  - [majentik/gemma-4-E4B-TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-E4B-TurboQuant-MLX-2bit) -- TurboQuant MLX 2-bit variant
 
137
 
138
  | Variant | Runtime | Approx size | Use case |
139
  |---|---|---|---|
 
140
  | [RotorQuant-GGUF-IQ4_XS](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-IQ4_XS) | llama.cpp | ~3.4 GB | Lossy 4-bit, low-RAM CPU/edge |
141
  | [RotorQuant-GGUF-Q2_K](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q2_K) | llama.cpp | ~2.4 GB | Lossy, low-RAM CPU/edge |
142
  | [RotorQuant-GGUF-Q3_K_M](https://huggingface.co/majentik/gemma-4-e4b-rotorquant-gguf-Q3_K_M) | llama.cpp | ~3.1 GB | Smaller 3-bit, CPU-friendly |
 
149
  | [TurboQuant](https://huggingface.co/majentik/gemma-4-e4b-turboquant) | runtime modifier | n/a | KV-cache root (weight-agnostic) |
150
  | [TurboQuant-MLX-2bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-2bit) | mlx-lm | ~1.3 GB | Apple Silicon, smallest |
151
  | [TurboQuant-MLX-4bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-4bit) | mlx-lm | ~2.5 GB | Apple Silicon balanced |
152
+ | [TurboQuant-MLX-8bit](https://huggingface.co/majentik/gemma-4-e4b-turboquant-mlx-8bit) | mlx-lm | ~4.7 GB | Apple Silicon reference |