Atomic Chat self-quantized google/gemma-4-E4B-it ladder (imatrix, calibration_datav3)

.gitattributes

@@ -33,3 +33,17 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+gemma4-e4b-it-UD-Q4_K_XL.gguf filter=lfs diff=lfs merge=lfs -text
+hero.png filter=lfs diff=lfs merge=lfs -text
+mmproj-gemma4-e4b-it-f16.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,141 @@
+---
+license: apache-2.0
+license_link: https://ai.google.dev/gemma/docs/gemma_4_license
+thumbnail: https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/hero.png
+base_model:
+- google/gemma-4-E4B-it
+base_model_relation: quantized
+quantized_by: AlexAtomic
+pipeline_tag: image-text-to-text
+library_name: gguf
+tags:
+- atomic-chat
+- gemma
+- gemma4
+- google
+- gguf
+- imatrix
+- quantized
+- llama.cpp
+---
+
+<center>
+
+<div style="display:flex; justify-content:center; align-items:center; gap:10px; flex-wrap:wrap;">
+<a href="https://atomic.chat"><img src="https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/pill_atomic_v3.png" alt="Atomic Chat" width="186"></a>
+<a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/pill_discord_v3.png" alt="Join Discord" width="184"></a>
+<a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/pill_github_v3.png" alt="GitHub" width="141"></a>
+</div>
+
+<br/>
+
+<img src="https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/hero.png" alt="Gemma 4 E4B" style="width:100%; max-width:100%; height:auto; margin-bottom:0.6em;"/>
+
+<div style="display:flex; justify-content:center; gap:0.5em;">
+<a href="https://huggingface.co/google/gemma-4-E4B-it"><strong>Base model: google/gemma-4-E4B-it</strong></a>
+</div>
+</center>
+
+**Gemma 4 E4B**, self-quantized to GGUF by [Atomic Chat](https://atomic.chat). Built straight from Google's original weights with a per-tensor importance matrix. Runs fully offline.
+
+## Highlights
+
+- **Natively multimodal** — handles text, image, and audio input and generates text output.
+- **4.5B effective parameters (8B with embeddings)** — the "E" stands for "effective", using Per-Layer Embeddings (PLE) for on-device efficiency.
+- **128K-token context window** built on a hybrid local/global attention mechanism.
+- **Built-in thinking mode** — configurable step-by-step reasoning, triggered with the `<|think|>` token.
+- **Native function calling** for structured tool use and agentic workflows.
+- **Multilingual** — out-of-the-box support for 35+ languages, pre-trained on 140+ languages.
+
+> [!NOTE]
+> These GGUFs are **self-quantized from the original weights**, not a repack. The importance matrix keeps low-bit quants closer to the full-precision model.
+
+> [!IMPORTANT]
+> Always pass `--jinja` so the **Gemma 4 E4B chat template** is applied. Without it the model can emit malformed turns.
+
+## Model Overview
+
+| Property | Value |
+|---|---|
+| Base model | `google/gemma-4-E4B-it` |
+| Parameters | 4.5B effective (8B with embeddings); uses Per-Layer Embeddings (PLE) |
+| Layers | 42 |
+| Context length | 128K tokens |
+| Vocabulary | 262K |
+| Modalities | Text, Image, Audio |
+| Architecture | Dense, hybrid local sliding-window (512) + global attention with p-RoPE |
+| This repo | GGUF quants (imatrix) + vision mmproj |
+
+> [!NOTE]
+> Gemma 4 E4B is multimodal. This repo ships the **`mmproj-gemma4-e4b-it-f16.gguf`** vision projector. With `-hf` it is pulled automatically; otherwise pass `--mmproj`. Use `llama-mtmd-cli` or `llama-server` to feed images.
+
+<img src="https://huggingface.co/AlexAtomic/gemma4-e4b-it-GGUF/resolve/main/benchmark.png" alt="Gemma 4 E4B benchmark scores" style="width:100%; max-width:760px;"/>
+
+Scores are Google's published results for the base `google/gemma-4-E4B-it`. Quantization preserves the large majority of this; `Q4_K_M` and up sit within a point or two of full precision.
+
+
+## Choosing a quant
+
+| Quant | Size | Notes |
+|---|---|---|
+| `Q2_K` | 4.4 GB | Smallest. Minimal RAM, clear quality drop. |
+| `IQ3_M` | 4.7 GB | Beats Q3 at similar size thanks to imatrix. Best low-RAM pick. |
+| `Q3_K_M` | 0.1 GB | Low quality but usable. |
+| `Q3_K_L` | 5.0 GB | A step above Q3_K_M. |
+| `IQ4_XS` | 5.1 GB | Excellent quality for size. Recommended low-bit. |
+| `Q4_K_S` | 0.1 GB | Compact Q4, fast. |
+| **`Q4_K_M`** | 5.3 GB | **Recommended default. Best balance of size, speed and quality.** |
+| **`UD-Q4_K_XL`** | 6.2 GB | **Dynamic. Embeddings and output kept at Q8_0 for higher quality at a Q4 footprint.** |
+| `Q5_K_S` | 5.7 GB | Higher quality. |
+| `Q5_K_M` | 5.8 GB | Higher quality, low loss. |
+| `Q6_K` | 6.2 GB | Near lossless. |
+| `Q8_0` | 8.0 GB | Effectively lossless, reference quality. |
+
+> [!TIP]
+> Pick the largest file that fits your (V)RAM with room for context. `Q4_K_M` or `UD-Q4_K_XL` is the sweet spot for most setups; `Q6_K` or `Q8_0` for maximum fidelity.
+
+## Get started
+
+Run Gemma 4 E4B locally with:
+
+- **[Atomic Chat](https://atomic.chat):** the easiest path. Open the app, search `AlexAtomic/gemma4-e4b-it-GGUF`, pick a quant, hit **Use this model**.
+- **llama.cpp:** `llama-server -hf AlexAtomic/gemma4-e4b-it-GGUF:Q4_K_M --jinja -c 8192`
+- **Ollama:** `ollama run hf.co/AlexAtomic/gemma4-e4b-it-GGUF:Q4_K_M`
+- **LM Studio / Jan:** search the repo id, download any quant.
+
+## Best practices
+
+| Parameter | Value |
+|---|---|
+| temperature | 1.0 |
+| top_p | 0.95 |
+| top_k | 64 |
+
+Google's standardized sampling configuration recommended across all use cases.
+
+## Run in llama.cpp
+
+```bash
+git clone https://github.com/ggerganov/llama.cpp
+cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON
+cmake --build llama.cpp/build --config Release -j --target llama-cli llama-server
+```
+
+```bash
+./llama.cpp/build/bin/llama-server \
+    -hf AlexAtomic/gemma4-e4b-it-GGUF:UD-Q4_K_XL \
+    --jinja -ngl 99 -c 8192 -fa on
+```
+
+## How these were made
+
+1. Download `google/gemma-4-E4B-it` (original weights).
+2. Convert to f16 GGUF with [llama.cpp](https://github.com/ggerganov/llama.cpp).
+3. Build an importance matrix over `calibration_datav3` (100 chunks).
+4. Quantize the full ladder with `--imatrix`.
+5. `UD-Q4_K_XL` additionally pins the token-embedding and output tensors to `Q8_0`.
+
+## License
+
+Original model by Google DeepMind, released under the Apache 2.0 license. Quantized by Atomic Chat.
+

gemma4-e4b-it-IQ3_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5408286c230d253d831d208651d79c1d68dcae88d995721415f975825e67c078
+size 4714696192

gemma4-e4b-it-IQ4_XS.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee912d0c765ac8480b8aa3904bc02b8e0564f93124988d5af7a3effa15382ac4
+size 5070956032

gemma4-e4b-it-Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12dd2286d2f05155cd14e1ac32e71f1b06065b8df1990f80b368ea95936d404a
+size 4401321472

gemma4-e4b-it-Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8ecbeba83958494f61254434217e4ed5fbd5ae8eb4a195c65b59a58179845928
+size 5021281792

gemma4-e4b-it-Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c78bdf973f56cacec724525c009f0ee731f1503eba815fc03e0c69fdc99a2da
+size 70886560

gemma4-e4b-it-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf4b60853354d05a8fc21aa7bb9e1f292a7e74c328331d7229499913b9d77dbd
+size 5335291392

gemma4-e4b-it-Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c78bdf973f56cacec724525c009f0ee731f1503eba815fc03e0c69fdc99a2da
+size 70886560

gemma4-e4b-it-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e28ec8266d0aafea0d2bae333e17e2fd9e43ea0e012454ef81ddccf329314a6
+size 5762913792

gemma4-e4b-it-Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:838c9b56271b09afc67512ebb11d01b5fb9e8becccb5836d6e048db7d8fd243e
+size 5685970432

gemma4-e4b-it-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ce1c3a87ca8e877f82da9d8af8bbc1726199a8f5e8482929ab2e79a20af509b
+size 6217262592

gemma4-e4b-it-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e68160d4c432e12b83ef42d9f82b450c394c308c7f828d37dac17e54ff02d94b
+size 8031241728

gemma4-e4b-it-UD-Q4_K_XL.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:43fb5fb7958bc169e49c9ccf570a253431752432a163f1e9a7efd8881f11c624
+size 6180443648

mmproj-gemma4-e4b-it-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b26ce301094fdc470b658b48061fc1a7c2535769619865c4f5031927a6374a6
+size 991552256