gemma-4-31B-v2-MLX-4bit

A 4-bit MLX build of an abliterated Gemma 4 31B for Apple Silicon. One repo, three ways to run it:

• Plain chat in LM Studio or mlx-lm, about 15 tok/s. Vision weights are stripped automatically.
• Fast text with MTP speculative decoding: up to 37.1 tok/s.
• Image understanding through mlx-vlm.

Everything except the vision tower is 4-bit (group size 64). Perplexity 59.80 vs 59.96 bf16. About 18 GB on disk, fits a 48 GB Mac.

Quick start (gets you the 31.8 tok/s config)

huggingface-cli download jdfelo/gemma-4-31B-v2-MLX-4bit run_mac.sh chat.html patch_mlx_vlm_fast_verify.py --local-dir .
bash run_mac.sh

The script installs mlx-vlm, applies the fast-verify patch, installs the fast mlx wheel from this repo if your machine can use it (Python 3.11 + macOS 26), downloads the model and drafter, starts an OpenAI-compatible server on :8080, and opens a browser chat. It picks the right block size for whichever mlx you end up on.

DRAFT=0 bash run_mac.sh   # dense only, no drafter
BLOCK=3 bash run_mac.sh   # force a block size (auto by default)

Apple Silicon only.

Speed (MacBook Pro M5 Pro, 48 GB)

Config	tok/s	speedup
dense 4-bit	15.0	1.0x
MTP, stock pip mlx 0.31.2	20.4	1.4x
MTP + fast-verify patch	24.6	1.6x
MTP + patch + fast mlx wheel	31.8	2.1x

Output is byte-identical to dense greedy at temperature 0 in every config (verified). Dense is bandwidth-bound (17.3 GB of weights per token, ~276 GB/s measured, ≈16 tok/s ceiling); everything past it comes from speculative decoding.

The two pieces behind the jump from 20 to 32, both in this repo:

1. patch_mlx_vlm_fast_verify.py. mlx-vlm's Gemma 4 is missing a hook its own MTP loop looks for, so every verify round did K+1 sequential lm_head projections with a blocking sync each. The patch adds the 9-line hook: one batched projection, one sync, same logits. Idempotent, --revert undoes it.
2. wheels/mlx-0.32.0.dev*.whl. A prebuilt mlx with the qmv_wide kernel (PR #3764, merged upstream but not yet on PyPI). Stock 0.31.2 re-reads the whole weight matrix once per verify row; qmv_wide reads it once per round. Verify at block 4 drops from 142 ms to 95 ms. Built for Python 3.11 on macOS 26 (Apple Silicon). On other setups, either build mlx from source (pip wheel git+https://github.com/ml-explore/mlx --no-deps, after xcodebuild -downloadComponent MetalToolchain) or skip it and stay at 24.6.

Doing it by hand

pip install -U "git+https://github.com/Blaizzy/mlx-vlm.git"
python patch_mlx_vlm_fast_verify.py
pip install --force-reinstall --no-deps "$(huggingface-cli download jdfelo/gemma-4-31B-v2-MLX-4bit wheels/mlx-0.32.0.dev20260702+e9463bb-cp311-cp311-macosx_26_0_universal2.whl)"

python -m mlx_vlm server --host 127.0.0.1 --port 8080 \
  --model jdfelo/gemma-4-31B-v2-MLX-4bit \
  --draft-model mlx-community/gemma-4-31B-it-assistant-bf16 \
  --draft-kind mtp --draft-block-size 4

Use --draft-block-size 3 if you stayed on stock mlx 0.31.2. One-off generation works the same way with python -m mlx_vlm generate ... --prompt "..." --temperature 0.

Dense in LM Studio: drop this repo into ~/.lmstudio/models/jdfelo/, load it, chat. LM Studio can't run the MTP drafter, so use mlx-vlm for the fast path.

Dense with mlx-lm:

pip install -U mlx-lm
mlx_lm.generate --model jdfelo/gemma-4-31B-v2-MLX-4bit --prompt "Hi" --max-tokens 128

Tip: sudo sysctl iogpu.wired_limit_mb=36864 keeps the weights wired on a 48 GB Mac. Ollama can't run this (it wants GGUF, these are MLX weights).

Responsible use

This is an abliterated, uncensored model with the usual safety refusals removed. You are responsible for how you use it. The underlying Gemma components remain covered by Google's Gemma Terms.