README.md

---
language: en
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
license: other
license_name: kimi-k3
license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
base_model: moonshotai/Kimi-K3
base_model_relation: quantized
---

# kernelpool/Kimi-K3-2bit-UVMAX

Mixed-precision (UVMAX) quantization of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3).

## What is UVMAX?

UVMAX is a mixed-precision scheme: bit widths are assigned per tensor class
from measured round-trip quantization error, rather than uniformly.

| Tensor class | Bits | Parameters | Size | Share |
|---|---|---|---|---|
| Expert FFNs (routed, latent space) | 2 (gs 128) | 2.72 T | 713.2 GiB | 93.8% |
| Shared experts, MoE latent projections, dense MLP | 8 (gs 64) | 17.5 B | 17.4 GiB | 2.3% |
| Attention (KDA + MLA, all projections) | 6 (gs 64) | 36 B | 27.4 GiB | 3.6% |
| Embeddings, `lm_head` | 4 (gs 64) | 2.4 B | 1.2 GiB | 0.2% |
| MoE routers | 8 (gs 64) | 0.6 B | 0.6 GiB | 0.1% |
| Vision tower + projector (unquantized bf16) | — | 0.4 B | 0.8 GiB | 0.1% |
| Norms, AttnRes projections, gate params (unquantized) | — | — | 0.1 GiB | <0.1% |

## Use with mlx

This model requires Kimi K3 support from
[mlx-lm PR #1626](https://github.com/ml-explore/mlx-lm/pull/1626), which has
not yet been merged. Until it is included in an mlx-lm release, install
mlx-lm from the PR branch:

```bash
pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1626/head
pip install tiktoken
```

```python
from mlx_lm import load, generate

model, tokenizer = load(
    "kernelpool/Kimi-K3-2bit-UVMAX",
    tokenizer_config={"trust_remote_code": True},
    trust_remote_code=True,
)

prompt = "hello"

messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, verbose=True)
```