File size: 2,242 Bytes
8fd6024 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 | ---
base_model: google/gemma-4-12B
license: gemma
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
- quantized
- gemma
- gemma4
- image-text-to-text
- apple-silicon
- rotorquant
---
# gemma-4-12B — RotorQuant MLX 8bit
[`google/gemma-4-12B`](https://huggingface.co/google/gemma-4-12B) @ `023679ed352de9bb66cc873c9009ce3482585c08` quantized pack, published as `majentik/gemma-4-12B-RotorQuant-MLX-8bit`.
## Method
MLX quantization via mlx-vlm 0.6.3 (8bit, group_size 64); vision + audio towers retained in BF16 (not quantized).
## Release line
Released under the **RotorQuant** line. RotorQuant and TurboQuant are this project's release labels for this pack, not distinct quantization algorithms — both brand repos for a given tier carry byte-identical weights, produced once and published under two names. No brand-specific speedup is claimed or measured for either label.
## Modality
This pack is `image-text-to-text` capable: the vision and audio towers ship in BF16 alongside the quantized text tower, so image (and audio) inputs are supported end to end via `mlx-vlm`.
**Base-model note:** this is the raw completion model (not instruction-tuned). It is not instruction-aligned for multimodal Q&A — expect free-form continuation behavior rather than chat-style image description, even though the vision/audio towers are present.
**mlx-vlm suppress-tokens note:** the base model's `generation_config.json` carries `suppress_tokens` for six multimodal placeholder token ids so that `transformers` generation is clean. `mlx-vlm` does not honor `suppress_tokens` automatically, so text-only generation with `mlx-vlm` can leak placeholder tokens (observed as a degenerate `A<image|>A<image|>...` loop in smoke testing) unless you suppress them yourself:
```python
from mlx_vlm import load, generate
model, processor = load("majentik/gemma-4-12B-RotorQuant-MLX-8bit")
suppress = {255999: -1e9, 256000: -1e9, 258880: -1e9, 258881: -1e9, 258882: -1e9, 258883: -1e9}
output = generate(
model, processor, prompt,
max_tokens=256,
logit_bias=suppress,
)
```
## License
Governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). See the upstream repo for the full license text.
|