Add oMLX oQ variants as build options + README
Browse files
README.md
CHANGED
|
@@ -76,6 +76,51 @@ For image prompting, add `--image /path/to/image.png` to the generation command.
|
|
| 76 |
Use current MLX-VLM documentation for image, audio, video, and chat-template
|
| 77 |
options.
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
## License
|
| 80 |
|
| 81 |
Gemma 4 is Apache-2.0. This derivative package uses the Apache-2.0 license
|
|
|
|
| 76 |
Use current MLX-VLM documentation for image, audio, video, and chat-template
|
| 77 |
options.
|
| 78 |
|
| 79 |
+
## Quantized variants (oMLX oQ)
|
| 80 |
+
|
| 81 |
+
This FP16 package is the canonical source for building oMLX oQ quantizations
|
| 82 |
+
on Apple Silicon. The build kit is included in this directory:
|
| 83 |
+
|
| 84 |
+
- `build_omlx_variants.py` — builds `oQ4/oQ6/oQ8` and `oQ4e/oQ6e/oQ8e`.
|
| 85 |
+
- `omlx_build_matrix.json` — the variant matrix the script mirrors.
|
| 86 |
+
|
| 87 |
+
### Available options
|
| 88 |
+
|
| 89 |
+
| Variant | Family | oQ level | Notes |
|
| 90 |
+
| --- | --- | --- | --- |
|
| 91 |
+
| `gemma4-e2b-Snowfox-MLX-FP16` | fp16 | — | shipped in this repo |
|
| 92 |
+
| `gemma4-e2b-Snowfox-MLX-oQ4` | oQ | 4 | GGUF `Q4_K_M` analogue |
|
| 93 |
+
| `gemma4-e2b-Snowfox-MLX-oQ6` | oQ | 6 | GGUF `Q6_K` analogue |
|
| 94 |
+
| `gemma4-e2b-Snowfox-MLX-oQ8` | oQ | 8 | near-lossless |
|
| 95 |
+
| `gemma4-e2b-Snowfox-MLX-oQ4e` | oQe | 4 | enhanced (sensitivity-calibrated) |
|
| 96 |
+
| `gemma4-e2b-Snowfox-MLX-oQ6e` | oQe | 6 | enhanced |
|
| 97 |
+
| `gemma4-e2b-Snowfox-MLX-oQ8e` | oQe | 8 | enhanced |
|
| 98 |
+
|
| 99 |
+
The `oQ*` variants are build-to-order on Apple Silicon (see below); only the
|
| 100 |
+
FP16 weights are stored in this repository.
|
| 101 |
+
|
| 102 |
+
```bash
|
| 103 |
+
# On Apple Silicon with the oMLX DMG installed:
|
| 104 |
+
/Applications/oMLX.app/Contents/MacOS/python3 build_omlx_variants.py \
|
| 105 |
+
--source . --output-root .. --no-smoke-test --dry-run
|
| 106 |
+
|
| 107 |
+
# Real run (quantizes + smoke-tests each variant):
|
| 108 |
+
/Applications/oMLX.app/Contents/MacOS/python3 build_omlx_variants.py \
|
| 109 |
+
--source . --output-root ..
|
| 110 |
+
```
|
| 111 |
+
|
| 112 |
+
Produces sibling directories `../gemma4-e2b-Snowfox-MLX-oQ4`, `-oQ6`, `-oQ8`,
|
| 113 |
+
`-oQ4e`, `-oQ6e`, `-oQ8e`. The `oQe` variants are the enhanced (sensitivity-
|
| 114 |
+
calibrated) family; `oQ4`/`oQ6` are the oMLX analogues of GGUF `Q4_K_M`/`Q6_K`.
|
| 115 |
+
|
| 116 |
+
> **Unverified for Gemma 4.** oMLX oQ was originally validated against
|
| 117 |
+
> `qwen3_5` (Hedgehog). Gemma 4 E2B's vision + audio towers have not been
|
| 118 |
+
> exercised through `quantize_oq_streaming`. If it cannot load the full
|
| 119 |
+
> multimodal model, retry with `--text-only` (language backbone only) or check
|
| 120 |
+
> the installed oMLX version's supported architectures. The smoke test uses
|
| 121 |
+
> `mlx_vlm` (not `mlx_lm`) because this is a multimodal model; skip it with
|
| 122 |
+
> `--no-smoke-test` if the installed `mlx-vlm` API differs.
|
| 123 |
+
|
| 124 |
## License
|
| 125 |
|
| 126 |
Gemma 4 is Apache-2.0. This derivative package uses the Apache-2.0 license
|