| --- |
| library_name: mlx |
| license: mit |
| pipeline_tag: text-generation |
| base_model: ricdomolm/mini-coder-4b |
| tags: |
| - mlx |
| - quantized |
| - mixed-precision |
| - 4bit |
| - 8bit |
| - optiq |
| - apple-silicon |
| - text-generation |
| - code |
| - agent |
| --- |
| |
| # mlx-community/mini-coder-4b-OptiQ-4bit |
|
|
| A 4-bit mixed-precision MLX quant produced by [mlx-optiq](https://mlx-optiq.com/) — the sensitivity-aware quantization toolkit for Apple Silicon. It beats stock uniform 4-bit on the six-metric Capability Score (45.83 vs 42.74, +3.09) and is ~2.2x more faithful to the original bf16 weights (KL 0.06 vs 0.13). |
|
|
| A 4-bit mixed-precision MLX quant of [ricdomolm/mini-coder-4b](https://huggingface.co/ricdomolm/mini-coder-4b) (a code/agent model fine-tuned from Qwen3-4B-Instruct-2507). Per-layer bit-widths come from a KL-divergence sensitivity pass on a [six-domain calibration mix](https://mlx-optiq.com/blog/calibration-mix) (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit. |
|
|
| ## Quantization details |
|
|
| | Property | Value | |
| |---|---| |
| | Predominant precision | 4-bit | |
| | Layers at 8-bit (sensitive) | 123 | |
| | Layers at 4-bit (robust) | 129 | |
| | Total quantized layers | 252 | |
| | Achieved bits per weight | 5.16 | |
| | Group size | 64 | |
| | Calibration mix | [six-domain mix](https://mlx-optiq.com/blog/calibration-mix) (40 samples × 6 domains) | |
| | Reference for sensitivity | bf16 | |
|
|
| We follow the same naming convention `llama.cpp` uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The few sensitive layers held at 8-bit make this build +0.7 GB larger than a stock uniform-4-bit quant (2.8 GB vs 2.1 GB) while recovering quality the uniform quant loses. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install mlx-lm |
| ``` |
|
|
| ```python |
| from mlx_lm import load, generate |
| |
| model, tokenizer = load("mlx-community/mini-coder-4b-OptiQ-4bit") |
| response = generate( |
| model, tokenizer, |
| prompt="Write a Python function to check if a string is a palindrome.", |
| max_tokens=512, |
| ) |
| ``` |
|
|
| It's a code/agent model — best paired with a lightweight agent harness / loop. For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server with structured/JSON output and server-side tool-call healing, hot-swap mounted adapters), install [`mlx-optiq`](https://mlx-optiq.com/): |
|
|
| ```bash |
| pip install mlx-optiq |
| ``` |
|
|
| ## Benchmarks |
|
|
| Six-metric Capability Score (mean of MMLU + GSM8K + IFEval + BFCL + HumanEval + HashHop). Apples-to-apples comparison against stock uniform 4-bit: |
|
|
| | Metric | OptiQ | Uniform 4-bit | Δ | |
| |---|---:|---:|---:| |
| | MMLU (5-shot, 1000 samples) | **69.9%** | 68.1% | +1.8 | |
| | GSM8K (1000 samples, CoT) | **59.6%** | 48.1% | +11.5 | |
| | IFEval (full set, strict) | **36.6%** | 38.4% | -1.8 | |
| | BFCL-V3 simple (200 calls) | **56.5%** | 47.5% | +9.0 | |
| | HumanEval (164 problems, pass@1) | **52.4%** | 54.3% | -1.8 | |
| | HashHop (long-context retrieval) | **0.0%** | 0.0% | +0.0 | |
| | **Capability Score** (mean of 6) | **45.83** | 42.74 | **+3.09** | |
| | KL vs bf16 (mean / p95) | **0.0571** / 0.2336 | 0.1277 / 0.4872 | — | |
| | On-disk size | 2.8 GB | 2.1 GB | +0.7 | |
|
|
| OptiQ's clearest wins are **GSM8K (+11.5)** and **tool-calling (BFCL +9.0)**, plus markedly better fidelity to the original weights. IFEval and HumanEval are within noise (~±7 pp at these sample sizes). HashHop is 0% for **both** quants: a genuine model-capability gap — the model has no long-context multi-hop retrieval ability (verified by inspecting outputs, it reasons in prose instead of walking the hash chain) — not a quantization artifact, which is why OptiQ and uniform-4 match there. |
|
|
| Every metric gets one equal vote; disk size is an honest second axis, not folded into the score. See the [eval-framework writeup](https://mlx-optiq.com/blog/eval-framework). |
|
|
| ## Links |
|
|
| - **Project website:** [mlx-optiq.com](https://mlx-optiq.com/) |
| - **Docs:** [mlx-optiq.com/docs](https://mlx-optiq.com/docs/) |
| - **PyPI:** [pypi.org/project/mlx-optiq](https://pypi.org/project/mlx-optiq/) |
| - **Eval framework:** [mlx-optiq.com/blog/eval-framework](https://mlx-optiq.com/blog/eval-framework) |
| - **Base model:** [ricdomolm/mini-coder-4b](https://huggingface.co/ricdomolm/mini-coder-4b) |
|
|
| ## License |
|
|
| MIT (inherits from the base model). |
|
|
