Uncensoring Gemma 4 12B with Heretic — Keeping Vision, Audio, and the Encoder-Free Design Intact

Arjun Reddy

junafinity

Follow

Research attribution: osmAPI Research Team Terv Student Research Team

Base model: google/gemma-4-12B-it Released artifacts: osmGemma-4-12B-uncensored (bf16 + 3 MLX quants)

TLDR

We abliterated (refusal-ablated) Google's brand-new Gemma 4 12B — the encoder-free unified multimodal model — using Heretic, dropping refusals from 99/100 to 12/100 on mlabonne/harmful_behaviors at a KL divergence of only 0.053 from the original. Crucially, we kept every vision and audio weight intact through both abliteration and quantization, and shipped a full-precision bf16 checkpoint plus three MLX quants for Apple Silicon. Along the way we fixed a bf16-on-MPS numerical bug in the refusal-direction search, and worked around the fact that the MLX vision tooling does not yet model this architecture's encoder-free image pathway.

Abstract

Gemma 4 12B is the first unified (encoder-free) Gemma: there is no separate SigLIP vision tower or audio encoder. Image patches and audio frames are projected directly into the language model and processed by the shared decoder. That design is wonderful for deployment size — and inconvenient for every tool that assumes a vision tower exists.

This report documents the full pipeline — download → abliterate → quantize → publish — entirely on a single 128 GB M3 Max MacBook Pro, including the platform bugs we had to clear, how we guaranteed the multimodal weights survived, where multi-token prediction (MTP) fits, and the honest limitations of MLX inference for this architecture today.

What makes Gemma 4 12B different

We confirmed the architecture directly from the checkpoint. The model is Gemma4UnifiedForConditionalGeneration (model_type: gemma4_unified), 11.95 B parameters, 48 decoder layers, 256 K context. Its tensor inventory is revealing:

There is no vision_tower.encoder.layers. There is no audio transformer. The vision_config/audio_config carry no num_hidden_layers at all — just patch sizes, soft-token counts, and projection dims. As Google's card puts it, the model brings "native audio and vision understanding… without the need for separate encoders." Preserving "vision and audio" therefore means preserving those 11 small projector tensors and keeping the shared decoder healthy.

Abliteration with Heretic

Heretic performs directional ablation: it finds the residual-stream directions most associated with refusal (from contrasting "harmful" and "harmless" prompt sets) and removes their projection from the weights, while an Optuna/TPE search co-optimizes two objectives — minimize refusals and minimize KL divergence from the original model.

Our run: 100 trials (40 random startup + TPE), 400 + 400 direction prompts (harmless_alpaca / harmful_behaviors), evaluation on harmful_behaviors test[:100], seed 42. Heretic only touches the text decoder's attn.o_proj and mlp.down_proj — it never sees the vision/audio projectors.

The Pareto front we selected (trial #55):

An 87.9 % reduction in refusals at a KL well under the 0.5 "capability damage" threshold.

The bf16-on-MPS bug that almost sank it

The optimizer kept reporting KL divergence: nan, which silently emptied the Pareto front. The cause is a nice trap. Gemma masks unused-vocabulary logits to -inf. Heretic computes refusal-vs-baseline KL as kl_div(logprobs, base_logprobs, log_target=True) = Σ exp(base)·(base − trial). At the masked positions, exp(-inf) = 0 and (base − trial) = (-inf) − (-inf) = nan, so the term becomes 0 · nan = nan. Casting to fp32 alone does not help — -inf survives the cast. The fix is to sanitize the logits before log_softmax:

logits = torch.nan_to_num(outputs.scores[0].float(), nan=0.0, posinf=1e4, neginf=-1e4)
logprobs = F.log_softmax(logits, dim=-1)

With finite logits the KL became well-defined (0.002–0.5 across trials) and the search converged.

The Apple Silicon gauntlet

Running a day-old architecture + a fresh tool on Metal surfaced a chain of fixable blockers:

Keeping image and sound

We verified preservation at the tensor level at every stage. The abliterated bf16 checkpoint matched the original's byte count to within 984 bytes — all 677 tensors retained, vision_config/audio_config intact.

Quantization was the harder part. mlx-vlm 0.5.0 ships a gemma4 module, but for this checkpoint it (a) resolves only under a gemma4_unified → gemma4 model-type remap, and (b) does not model the vision_embedder at all — loading rejects its 9 tensors as "unexpected." Rather than drop them (which would silently delete the vision pathway), we used a hybrid:

1. Let mlx-vlm quantize the text + projection weights (it already keeps multimodal modules at higher precision), loading non-strict so the 9 unmodeled tensors are skipped.
2. Re-insert the 9 vision_embedder tensors at bf16 into the quantized output, and restore model_type: gemma4_unified + the multimodal configs.

The result is weight-complete: every original tensor is present, the language model is quantized, and the vision patch-embedder is preserved losslessly. The moment mlx-vlm adds the encoder-free vision forward, these quants light up with no re-quantization.

Where MTP fits

Gemma 4 ships its multi-token-prediction / speculative head separately as google/gemma-4-12B-it-assistant (a ~0.4 B draft model) — it is not baked into the 12B weights. So there were no MTP tensors to preserve in-model; an abliterated + quantized draft companion can be published later and loaded alongside for speculative decoding without touching these repos. (Like our Flash Load work, we scope MTP/speculative layers out of this proof.)

Quantization: four artifacts

All four preserve the full multimodal weight set.

A note on bits-per-weight: the 262 K-token vocabulary embedding (~1 B params, kept high-precision) sets a hard floor on the average bpw — true ~3.7 bpw is not reachable on this model without quantizing the embeddings, which we judged too risky for quality.

Results

The headline, measured identically across the family:

MLX text generation is verified working (with the chat template + a small load shim); for example the 8-bit quant answers cleanly on an M3 Max. Full multimodal (image + audio + video) runs today via 🤗 transformers on the bf16 checkpoint.

Limitations

• Vision/audio inference in MLX is pending. The weights are preserved, but mlx-vlm 0.5.0 doesn't yet implement the encoder-free vision patch-embedder forward. Text works now; full multimodal works now only via transformers (bf16).
• bpw above nominal for the MXFP4 (7.6) and mixed (4.2) builds, per the vocab-embedding floor.
• Bounded benchmark. Refusal rate measured on 100 prompts; broader red-team coverage and capability benchmarking are future work.
• MTP/speculative layers are out of scope here.

Live artifacts

• 🤗 Models: the osmGemma-4-12B-uncensored family on osmAPI
• 🔓 Abliteration: Heretic
• 🍎 Run on a Mac: oMLX, vMLX, LM Studio, Ollama, mlx-vlm

Acknowledgements

Thank you to:

• osmAPI Research Team
• Terv Student Research Team
• Heretic by p-e-w for the abliteration engine
• Google DeepMind for the open-weights Gemma 4 release
• The Apple MLX team and mlx-vlm
• The broader local-inference community (oMLX, vMLX, LM Studio, Ollama, llama.cpp) pushing Apple-Silicon inference forward