| --- |
| language: en |
| pipeline_tag: text-generation |
| tags: |
| - mlx |
| library_name: mlx |
| base_model: |
| - moonshotai/Kimi-K2.6 |
| --- |
| |
| [Kimi K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) optimized to run _comfortably_ |
| on a Mac Studio M3 512. This is the smaller, compact version. Quality-first |
| version [here](https://huggingface.co/spicyneuron/Kimi-K2.6-MLX-3.6bit). |
|
|
| - A mixed-precision quant that balances speed, memory, and accuracy. |
| - 3-bit baseline with important layers at 8-bit and BF16. |
| - Fits into ~430 GB memory, leaving plenty of room to run a smaller, faster utility model (ex: Qwen 3.6 35B, Gemma 4 26B). |
| - This quant does not support image input. |
|
|
| # Usage |
|
|
| ```sh |
| # Start server at http://localhost:8080/v1/chat/completions |
| # Kimi K2.6 requires tiktoken + remote code for the tokenizer |
| uvx --from mlx-lm --with tiktoken \ |
| mlx_lm.server \ |
| --host 127.0.0.1 \ |
| --port 8080 \ |
| --trust-remote-code \ |
| --model spicyneuron/Kimi-K2.6-MLX-3.3bit |
| ``` |
|
|
| # Benchmarks |
|
|
| metric | 3.6 bit | 3.3 bit (this model) |
| --- | --- | --- |
| bpw | 3.578 | 3.331 |
| peak memory (1024/512) | 460.444 | 428.735 |
| prompt tok/s (1024) | 221.704 ± 0.057 | 223.613 ± 0.098 |
| gen tok/s (512) | 21.095 ± 0.070 | 21.363 ± 0.035 |
| kl mean | 0.022 ± 0.001 | 0.051 ± 0.002 |
| kl p95 | 0.053 ± 0.001 | 0.113 ± 0.002 |
| perplexity | 3.559 ± 0.021 | 3.550 ± 0.020 |
| hellaswag | 0.594 ± 0.022 | 0.590 ± 0.022 |
| piqa | 0.848 ± 0.016 | 0.852 ± 0.016 |
| winogrande | 0.670 ± 0.021 | 0.690 ± 0.021 |
|
|
| Tested on a Mac Studio M3 Ultra with: |
|
|
| ``` |
| mlx_lm.kld --baseline-model path/to/mlx-full-precision |
| mlx_lm.perplexity --sequence-length 512 --seed 123 |
| mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5 |
| mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 500 |
| mlx_lm.evaluate --tasks piqa --seed 123 --num-shots 0 --limit 500 |
| mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 500 |
| ``` |
|
|
| Note: |
|
|
| - `mlx_lm.kld` is approximate, based on `top_k` not full logits. Here's the [code](https://github.com/ml-explore/mlx-lm/pull/1146). |
| - Kimi K2.6 KL divergence calculated against the largest quant I could run locally (~490 GB), so real KL is higher. |
|
|
| # Methodology |
|
|
| Quantized with a [mlx-lm fork](https://github.com/ml-explore/mlx-lm/pull/922), |
| drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. |
| MLX quantization options differ from llama.cpp, but the principles are the |
| same: |
|
|
| - Sensitive layers like MoE routing, attention, and output embeddings get higher precision |
| - More tolerant layers like MoE experts get lower precision |
|
|
