Can we get a 6 bit of mlx-community/Ornith-1.0-9B-OptiQ-4bit
HEllo Optiq Team,
is it possible for you to produce a mlx-community/Ornith-1.0-9B-OptiQ-6bit? thanks a lot for all your help. I am looking for a quality first type quantization of the Ornith model but don't have the RAM to produce it myself.
Why would you need such model?
HEllo Optiq Team,
is it possible for you to produce a mlx-community/Ornith-1.0-9B-OptiQ-6bit? thanks a lot for all your help. I am looking for a quality first type quantization of the Ornith model but don't have the RAM to produce it myself.
I can do it for you :)
The 4-bit sometime skips instructions, sometimes It does think deeply as the 6-bit one, it did try 6-bit FLAT non-optiq and 8-bit flat (in streaming mode) and the results were much better. so I was trying to quantize my own 6-bit OPTIQ but no enough RAM to do it. So hope anyone in the OPTIQ team has time to quantize such a mode.
If there is an online paid service for quantization whether here in hugging face or somewhere else, would be very grateful if you let me know.
Why would you need such model?
The 4-bit sometime skips instructions, sometimes It does think deeply as the 6-bit one, it did try 6-bit FLAT non-optiq and 8-bit flat (in streaming mode) and the results were much better. so I was trying to quantize my own 6-bit OPTIQ but no enough RAM to do it. So hope anyone in the OPTIQ team has time to quantize such a mode.
If there is an online paid service for quantization whether here in hugging face or somewhere else, would be very grateful if you let me know.
Done: mlx-community/Ornith-1.0-9B-OptiQ-6bit. 117 layers at 8-bit, 133 at 6-bit, group size 64, bf16 vision sidecar kept. 9.06 GB total, of which 8.15 GB is the language tower.
The floor moves from 4 to 6 bits against the 4-bit quant, and that is the whole difference: 1.05 GB more on the language tower. Weight-level fidelity, dequantized against the bf16 source: 0.742% relative error on the 8-bit layers, 2.266% on the 6-bit, 1.771% overall. The 4-bit variant's 4-bit layers sit at 9.8% by its own card.
Two things to be straight about.
The bit allocation is structural (--method static), not measured. I started an exact KL sweep and stopped it. At candidate bits 6 and 8 the measured per-layer gap between the two is 3-5%, against roughly an order of magnitude for 4 vs 8, and on 32 GB the sweep has to reference a uniform-4-bit baseline that is worse than both candidates. The ranking would have been noise wearing a "measured" label. Full reasoning is on the card.
I did not measure the thing you actually reported. Skipped instructions and shallower reasoning are behavioural, and I ran no benchmark comparing this to the 4-bit on it. Higher bits and 2.3x lower reconstruction error make it plausible this helps; that is not evidence. Please tell me if it holds up for you.
One practical note: if 10.43 GB fits your machine, the plain 8-bit you said already worked well is only 1.4 GB above this one. It has no vision sidecar, so this is the better pick if you need image input, but for text alone it is worth trying first.
Image input needs mlx-optiq>=0.4.17. Earlier versions look for the sidecar only in the repo root and will not find it in optiq/.
Done: mlx-community/Ornith-1.0-9B-OptiQ-6bit. 117 layers at 8-bit, 133 at 6-bit, group size 64, bf16 vision sidecar kept. 9.06 GB total, of which 8.15 GB is the language tower.
The floor moves from 4 to 6 bits against the 4-bit quant, and that is the whole difference: 1.05 GB more on the language tower. Weight-level fidelity, dequantized against the bf16 source: 0.742% relative error on the 8-bit layers, 2.266% on the 6-bit, 1.771% overall. The 4-bit variant's 4-bit layers sit at 9.8% by its own card.
Two things to be straight about.
The bit allocation is structural (
--method static), not measured. I started an exact KL sweep and stopped it. At candidate bits 6 and 8 the measured per-layer gap between the two is 3-5%, against roughly an order of magnitude for 4 vs 8, and on 32 GB the sweep has to reference a uniform-4-bit baseline that is worse than both candidates. The ranking would have been noise wearing a "measured" label. Full reasoning is on the card.I did not measure the thing you actually reported. Skipped instructions and shallower reasoning are behavioural, and I ran no benchmark comparing this to the 4-bit on it. Higher bits and 2.3x lower reconstruction error make it plausible this helps; that is not evidence. Please tell me if it holds up for you.
One practical note: if 10.43 GB fits your machine, the plain 8-bit you said already worked well is only 1.4 GB above this one. It has no vision sidecar, so this is the better pick if you need image input, but for text alone it is worth trying first.
Image input needs
mlx-optiq>=0.4.17. Earlier versions look for the sidecar only in the repo root and will not find it inoptiq/.
OMG can't thank you enough, you folks are cool
Uploaded and verified from the Hub in a clean cache: text, image, and the bit arithmetic (U32 × 32/bits over all 250 tensors sums to exactly 8,952,741,888).
One correction to my note above, since it named the wrong culprit.
I said image input needs mlx-optiq>=0.4.17 because older versions look for the sidecar only in the repo root. That is true, but I had also concluded that optiq.vlm.sidecar.build_vision_sidecar could not handle this architecture at all, because its vision-tower prefix list has no visual. entry while qwen3_5 checkpoints store the tower under model.visual.. So I wrote my own builder.
That was wrong, and it was my fault, not the tool's. A bare pip install mlx-optiq had resolved to 0.2.18, not the current 0.4.17. The visual. prefix and the optiq/ sidecar lookup were both added upstream somewhere after 0.2.18. On 0.4.17 the stock builder works on this model with no changes.
I checked my replacement against the stock builder's output at the byte level rather than assuming: all 333 tensors bit-identical, same dtypes and shapes, 912,020,960 bytes of payload on both. The two files differ only in the order tensors are written inside the container, which shifts data_offsets and nothing else, since loaders address tensors by name. So the published sidecar is equivalent to what 0.4.17 produces.
Worth checking optiq.__version__ rather than trusting a fresh install, which is the actual lesson.
Also uploaded a build here: https://huggingface.co/codelion/Ornith-1.0-9B-OptiQ-6bit
Same 6/8 mixed precision, 8.0 GB, vision sidecar at bf16. Bit-widths were measured on Ornith itself (249 layers, KL against the bf16 reference) rather than transferred, and the full sweep ships in the repo as optiq/sensitivity.json so it can be re-quantized at another target without repeating the measurement.
Text and image input both verified on the finished artifact.