Instructions to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("lvalenty/gemma-4-e2b-it-text-mlx-4bit")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "lvalenty/gemma-4-e2b-it-text-mlx-4bit"

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "lvalenty/gemma-4-e2b-it-text-mlx-4bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "lvalenty/gemma-4-e2b-it-text-mlx-4bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "lvalenty/gemma-4-e2b-it-text-mlx-4bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

MLX LM

How to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "lvalenty/gemma-4-e2b-it-text-mlx-4bit"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "lvalenty/gemma-4-e2b-it-text-mlx-4bit"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "lvalenty/gemma-4-e2b-it-text-mlx-4bit",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use lvalenty/gemma-4-e2b-it-text-mlx-4bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "lvalenty/gemma-4-e2b-it-text-mlx-4bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default lvalenty/gemma-4-e2b-it-text-mlx-4bit

Run Hermes

hermes

Atomic Chat

gemma-4-e2b-it-text-mlx-4bit / README.md

lvalenty

Text-only PLE-safe MLX 4-bit from google/gemma-4-E2B-it-qat-q4_0-unquantized@6befbaca

3928aa7 verified 9 days ago

preview code

Raw

History Blame Contribute Delete

7.74 kB

	---
	library_name: mlx
	license: apache-2.0
	license_link: https://ai.google.dev/gemma/docs/gemma_4_license
	base_model: google/gemma-4-E2B-it-qat-q4_0-unquantized
	base_model_relation: quantized
	pipeline_tag: text-generation
	language:
	- en
	tags:
	- mlx
	- gemma4
	- gemma
	- text-only
	- qat
	- ple-safe
	- on-device
	---

	# gemma-4-e2b-it-text-mlx-4bit

	A text-only, PLE-safe, 4-bit MLX build of Gemma 4 E2B, derived from **Google's official
	quantization-aware-trained (QAT) checkpoint**. Purpose-built for on-device inference on
	iPhone/Mac via `mlx-swift-lm` / `mlx-lm`: the vision tower and audio encoder are removed at
	conversion, and the Per-Layer-Embedding (PLE) pathway is quantized according to Google's own
	mobile QAT scheme rather than uniformly.

	\| \| \|
	\|---\|---\|
	\| Base \| [`google/gemma-4-E2B-it-qat-q4_0-unquantized`](https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-unquantized) \|
	\| Base commit \| `6befbaca7398925921802abd1f277b495b78b738` \|
	\| Base lineage \| `google/gemma-4-E2B-it` → QAT pipeline → half-precision QAT weights \|
	\| Architecture \| `gemma4_text` (text-only; no `vision_config`, no `audio_config`) \|
	\| Precision \| 4-bit affine, group 64, with PLE-safe exclusions (see below) \|
	\| Bits per weight \| 4.559 \|
	\| Size on disk \| 2.49 GiB (2.67 GB), 4 files + tokenizer \|
	\| Parameters \| 4.629 B (text stack only) \|

	## Why this exists

	1. Text-only. Google publishes E2B as a single 10.2 GB multimodal shard — file-level
	exclusion of the towers is impossible against the upstream repo. Converting ourselves makes
	the download manifest text-only by construction.
	2. PLE-safe. E2B's Per-Layer Embeddings feed each decoder layer through scalar-scaled
	projections. Quantizing that pathway uniformly at 4 bits measurably degrades the model
	(numbers below). Google's own mobile QAT scheme keeps those projections at 8-bit or full
	precision; this build mirrors that.
	3. QAT-derived. Starting from Google's QAT weights rather than post-quantizing the plain
	bf16 checkpoint is, by a wide margin, the single largest quality factor measured.

	## What was removed

	All vision and audio parameters are dropped at conversion. The model cannot accept image or
	audio input; the tokenizer and chat template retain their media special tokens, which are inert.

	\| Dropped prefix \| Tensors \| Parameters \|
	\|---\|---\|---\|
	\| `model.vision_tower.*` \| (part of 1,411) \| \|
	\| `model.audio_tower.*` \| \| \|
	\| `model.embed_vision.*` \| \| \|
	\| `model.embed_audio.*` \| \| \|
	\| Total dropped \| 1,411 \| 0.4757 B (0.886 GiB bf16) \|
	\| Total kept (text stack incl. PLE, embeddings) \| 540 \| 4.6286 B (8.621 GiB bf16) \|

	Size math: 10.209 GB source shard → 8.621 GiB text stack in bf16 → 2.49 GiB after PLE-safe
	4-bit quantization. The Per-Layer-Embedding table alone (`embed_tokens_per_layer`,
	262,144 × 8,960 = 2.349 B parameters) is 46% of the whole model and ~53% of this artifact; it is
	the reason a text-only 4-bit E2B lands near 2.5 GB rather than the ~1.5 GB one would estimate
	from "2B-class model at 4 bits".

	## Quantization recipe

	Derived directly from `google/gemma-4-E2B-it-qat-mobile-ct`'s `quantization_config`
	(Google's own mobile QAT scheme), read off its `config_groups` and `ignore` list:

	\| Module \| Google mobile QAT \| This build \|
	\|---\|---\|---\|
	\| `model.per_layer_model_projection` \| ignored — full precision \| bf16, not quantized \|
	\| `layers.N.per_layer_input_gate` \| 8-bit \| 8-bit, group 64 \|
	\| `layers.N.per_layer_projection` \| 8-bit \| 8-bit, group 64 \|
	\| `model.embed_tokens_per_layer` (PLE table) \| 4-bit, group 256 \| 4-bit, group 64 (finer ⇒ lower error) \|
	\| `layers.N.self_attn.{q,k,v,o}_proj` \| 4-bit \| 4-bit, group 64 \|
	\| `layers.N.mlp.{gate,up,down}_proj` \| 4-bit (15 layers) / 2-bit (20 layers) \| 4-bit, group 64 (we do not take the 2-bit tier) \|
	\| `model.embed_tokens` (tied to lm_head) \| 2-bit channel \| 4-bit, group 64 (more conservative) \|
	\| all RMSNorms \| not quantized \| bf16 \|

	## Measured quality

	Perplexity on 16,384 tokens of long structured agent-context text (2,048-token chunks),
	all measured with the same `mlx-lm` build on the same machine:

	\| Model \| Perplexity \|
	\|---\|---\|
	\| bf16 text-only reference (this build's QAT source, unquantized) \| 21.13 \|
	\| this build — text-only, PLE-safe 4-bit \| 20.51 \|
	\| ablation: same QAT source, naive uniform 4-bit (no PLE exclusions) \| 21.36 \|
	\| `mlx-community/gemma-4-E2B-it-qat-4bit` (QAT, PLE at 4-bit, MLP at 8-bit) \| 22.29 \|
	\| `mlx-community/gemma-4-e2b-it-4bit` (non-QAT base, uniform 4-bit) \| 106.85 \|

	Two findings worth stating plainly:

	* QAT lineage dominates. Uniform 4-bit applied to the QAT checkpoint is fine (21.36);
	uniform 4-bit applied to the plain bf16 checkpoint is catastrophic (106.85). If you take one
	thing from this card: quantize the QAT release, not the base release.
	* PLE-safety is a real but second-order win. Protecting the scalar-scaled PLE projections
	moves 21.36 → 20.51 and puts the 4-bit build slightly below its own bf16 reference — expected
	behaviour for QAT weights, which are trained to sit on a 4-bit grid.

	Throughput on an Apple M5 Max (mlx-lm, greedy): ~197 tok/s decode on short prompts, ~84 tok/s
	decode with a 2,300-token prompt in context; prompt prefill ~29 tok/s cold on the first call.
	Generation was checked on 11 prompts spanning factual QA, arithmetic reasoning, strict formatting,
	JSON-only output, narrative, dialogue, code, summarization, negative constraints, and one
	2,300-token structured agent context — coherent throughout, no repetition collapse.

	## Usage

	```python
	from mlx_lm import load, generate

	model, tokenizer = load("<this-repo>")
	prompt = tokenizer.apply_chat_template(
	[{"role": "user", "content": "Why does a village need a granary?"}],
	add_generation_prompt=True, tokenize=False,
	)
	print(generate(model, tokenizer, prompt, max_tokens=256))
	```

	Requires `mlx-lm` with `gemma4_text` support (≥ 0.31). Text input only.

	## Reproducing

	```
	source : google/gemma-4-E2B-it-qat-q4_0-unquantized @ 6befbaca7398925921802abd1f277b495b78b738
	convert: mlx 0.32.0 / mlx-lm 0.31.3
	drop model.{vision_tower,audio_tower,embed_vision,embed_audio,multi_modal_projector}.*
	rename model.language_model.* -> model.*
	config = source config.json "text_config", model_type=gemma4_text
	quantize group_size=64 bits=4 mode=affine with the PLE-safe predicate above
	```

	## License and attribution

	This is a derivative of Google's Gemma 4. It is distributed under the Apache License 2.0,
	the license of the base model — see the
	[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Gemma is a family of open
	models built by Google DeepMind; the original model card, capability descriptions, evaluation
	results, and limitations are in the
	[base repository](https://huggingface.co/google/gemma-4-E2B-it-qat-q4_0-unquantized) and apply
	to this derivative.

	Users of this derivative are directed to Google's responsible-use guidance for Gemma, which the
	base model card carries forward:

	* [Responsible Generative AI Toolkit](https://ai.google.dev/responsible)
	* [Google's AI Principles](https://ai.google/principles/)

	Known limitations of the base model — including that quality and diversity of training data bound
	its capabilities, that it can generate false or misleading text, and that it should not be relied
	on for factual accuracy without verification — carry over unchanged. This build additionally
	removes all image and audio capability: any multimodal claim in the base model card does not
	apply here.

	Attribution: Gemma 4 © Google DeepMind. Technical report: [arXiv:2607.02770](https://arxiv.org/abs/2607.02770).