File size: 1,923 Bytes
72d65b9 edb5113 72d65b9 edb5113 72d65b9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
language: en
library_name: mlx
pipeline_tag: image-text-to-text
tags:
- mlx
license: other
license_name: kimi-k3
license_link: https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
base_model: moonshotai/Kimi-K3
base_model_relation: quantized
---
# kernelpool/Kimi-K3-2bit-UVMAX
Mixed-precision (UVMAX) quantization of [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3).
## What is UVMAX?
UVMAX is a mixed-precision scheme: bit widths are assigned per tensor class
from measured round-trip quantization error, rather than uniformly.
| Tensor class | Bits | Parameters | Size | Share |
|---|---|---|---|---|
| Expert FFNs (routed, latent space) | 2 (gs 128) | 2.72 T | 713.2 GiB | 93.8% |
| Shared experts, MoE latent projections, dense MLP | 8 (gs 64) | 17.5 B | 17.4 GiB | 2.3% |
| Attention (KDA + MLA, all projections) | 6 (gs 64) | 36 B | 27.4 GiB | 3.6% |
| Embeddings, `lm_head` | 4 (gs 64) | 2.4 B | 1.2 GiB | 0.2% |
| MoE routers | 8 (gs 64) | 0.6 B | 0.6 GiB | 0.1% |
| Vision tower + projector (unquantized bf16) | — | 0.4 B | 0.8 GiB | 0.1% |
| Norms, AttnRes projections, gate params (unquantized) | — | — | 0.1 GiB | <0.1% |
## Use with mlx
This model requires Kimi K3 support from
[mlx-lm PR #1626](https://github.com/ml-explore/mlx-lm/pull/1626), which has
not yet been merged. Until it is included in an mlx-lm release, install
mlx-lm from the PR branch:
```bash
pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1626/head
pip install tiktoken
```
```python
from mlx_lm import load, generate
model, tokenizer = load(
"kernelpool/Kimi-K3-2bit-UVMAX",
tokenizer_config={"trust_remote_code": True},
trust_remote_code=True,
)
prompt = "hello"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, verbose=True)
```
|