| --- |
| license: mit |
| base_model: |
| - moonshotai/Kimi-K3 |
| library_name: transformers |
| tags: |
| - compressed-tensors |
| - LLM Compressor |
| - vLLM |
| --- |
| |
| # RedHatAI/Kimi-K3-NVFP4 |
|
|
| This is a quantized version of `moonshotai/Kimi-K3` with MoE layers quantized to NVFP4 for accelerated inference. |
|
|
| ## Usage |
|
|
| This model is intended for deployment with vLLM. You can serve the model using |
|
|
| ```bash |
| vllm serve RedHatAI/Kimi-K3-NVFP4 \ |
| --tensor-parallel-size 8 \ |
| --trust_remote_code \ |
| --load-format instanttensor \ |
| --reasoning-parser kimi_k3 \ |
| --language-model-only # optional |
| ``` |
|
|
| May require https://github.com/vllm-project/vllm/pull/50500 to run |
|
|
| ## Creation Process |
| This model was compressed using LLM Compressor. For more information, please contact Kyle Sayers via the [vLLM Developers Slack](https://communityinviter.com/apps/vllm-dev/join-vllm-developers-slack) or ksayers@redhat.com. |
|
|
| ## Evaluation ## |
| | Benchmark | `moonshotai/Kimi-K3` | `RedHatAI/Kimi-K3-NVFP4` |
| | - | - | -| |
| | GPQA | 93.5 | 91.0 | |
|
|
| ```bash |
| inspect eval hf/Idavidrein/gpqa/diamond \ |
| --model vllm/RedHatAI/Kimi-K3-NVFP4 \ |
| --reasoning-effort high \ |
| --model-base-url http://localhost:8000/v1 |
| ``` |
|
|
