Quantization ladder
Variants produced by hy_embodied_mlx/quantize.py from the bf16 MLX
conversion: affine quantization, group size 64, at 8/6/5/4 bits plus a
3-bit probe kept strictly experimental. Policy: the 448 decoder linears
(text and _v paths) and the tied embedding table are quantized — 449
modules per variant; the vision tower, merger, and all norms stay bf16
(the ViT fc2 input dim, 4304, is not divisible by 64, and the vision stack
is a small fraction of total bytes). Every per-module decision is written
to quantization_manifest.json in the variant directory; nothing is
skipped silently.
Measured on M3 Max, 36 GB, macOS 26.5.2, mlx 0.32.0
(oracle/smoke_quant.py, full outputs in docs/smoke/). Decode rate is
the median of 5 runs of 64 greedy tokens after a text prefill; image
prefill is the 333-token synthetic-image prompt (300 vision tokens)
including the ViT forward.
| variant | weights | decode tok/s | image prefill (s) |
|---|---|---|---|
| bf16 | 7.05 GiB | 66.8 | 0.18 |
| 8-bit | 4.14 GiB | 106.9 | 0.18 |
| 6-bit | 3.36 GiB | 124.3 | 0.18 |
| 5-bit | 2.98 GiB | 136.9 | 0.18 |
| 4-bit | 2.59 GiB | 157.6 | 0.18 |
| 3-bit (experimental) | 2.20 GiB | 171.2 | 0.18 |
Image prefill is flat across tiers because it is dominated by the ViT, which stays bf16 everywhere.
The smoke records also carry greedy token agreement against the bf16 oracle over 64 steps. That number falls quickly at lower bits (a single flipped token cascades in greedy decoding) and is not a quality metric; outputs remain coherent down to 3-bit on the smoke prompts. Quality per tier is measured by pointing accuracy in the eval harness, not by token agreement.