File size: 1,159 Bytes
be457b5 71eed69 be457b5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | ---
license: mit
base_model:
- moonshotai/Kimi-K3
library_name: transformers
tags:
- compressed-tensors
- LLM Compressor
- vLLM
---
# RedHatAI/Kimi-K3-NVFP4
This is a quantized version of `moonshotai/Kimi-K3` with MoE layers quantized to NVFP4 for accelerated inference.
## Usage
This model is intended for deployment with vLLM. You can serve the model using
```bash
vllm serve RedHatAI/Kimi-K3-NVFP4 \
--tensor-parallel-size 8 \
--trust_remote_code \
--load-format instanttensor \
--reasoning-parser kimi_k3 \
--language-model-only # optional
```
May require https://github.com/vllm-project/vllm/pull/50500 to run
## Creation Process
This model was compressed using LLM Compressor. For more information, please contact Kyle Sayers via the [vLLM Developers Slack](https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack) or ksayers@redhat.com.
## Evaluation ##
| Benchmark | `moonshotai/Kimi-K3` | `RedHatAI/Kimi-K3-NVFP4`
| - | - | -|
| GPQA | 93.5 | 91.0 |
```bash
inspect eval hf/Idavidrein/gpqa/diamond \
--model vllm/RedHatAI/Kimi-K3-NVFP4 \
--reasoning-effort high \
--model-base-url http://localhost:8000/v1
```
|