README.md

---
pipeline_tag: text-generation
base_model:
- moonshotai/Kimi-K2.7-Code
license: other
license_name: modified-mit
license_link: https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/LICENSE
library_name: Model Optimizer
tags:
- ModelOpt
- Kimi-K2.7-Code
- quantized
- FP4
- fp4
- NVFP4
---

# Kimi-K2.7-Code-NVFP4

> 🚀 **Available now** — try this model at **[cogito.decart.ai](https://cogito.decart.ai)** and via **[OpenRouter](https://openrouter.ai)**.

## Description:
NVFP4 quantized version of [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code), quantized with [NVIDIA Model Optimizer](https://github.com/NVIDIA/Model-Optimizer). The routed-expert linear layers are quantized to NVFP4 (4-bit float, block size 16) with an FP8 KV cache; attention (MLA), shared experts, the layer-0 dense MLP, `lm_head`, and the vision tower / `mm_projector` remain BF16 — the same precision split as [nvidia/Kimi-K2.6-NVFP4](https://huggingface.co/nvidia/Kimi-K2.6-NVFP4). Ready for inference with vLLM on NVIDIA Blackwell.

This is a community reproduction produced by Decart; it is not affiliated with or endorsed by NVIDIA or Moonshot AI.

## Third-Party Community Consideration
This model is not owned or developed by NVIDIA or Decart's base-model providers. It is built to a third-party's requirements; see the non-Decart [Kimi-K2.7-Code Model Card](https://huggingface.co/moonshotai/Kimi-K2.7-Code).

### License/Terms of Use:
Use of this model is governed by the license of the base model, [moonshotai/Kimi-K2.7-Code](https://huggingface.co/moonshotai/Kimi-K2.7-Code/blob/main/LICENSE) (Modified MIT).

## Model Architecture:
**Architecture Type:** Transformers <br>
**Network Architecture:** DeepSeek-V3 (MLA attention, 384 routed experts + 1 shared expert, 61 layers), wrapped with a vision tower + mm_projector (`KimiK25ForConditionalGeneration`) <br>
**Number of Model Parameters:** ~1T total / ~32B activated <br>

## Input:
**Input Type(s):** Text (Image/Video per base model) <br>
**Input Format(s):** String <br>
**Other Properties Related to Input:** Long context per base model <br>

## Output:
**Output Type(s):** Text <br>
**Output Format:** String <br>

## Software Integration:
**Supported Runtime Engine(s):** vLLM <br>
**Supported Hardware Microarchitecture Compatibility:** NVIDIA Blackwell <br>
**Preferred Operating System(s):** Linux <br>

## Post Training Quantization
This model was obtained by converting and quantizing the routed-expert weights and activations of Kimi-K2.7-Code from its native INT4 (compressed-tensors `pack-quantized`) → BF16 → NVFP4, ready for inference with vLLM. Only the weights and activations of the linear operators within the MoE transformer blocks are quantized.

- **Recipe:** `general/ptq/nvfp4_experts_only_mse-kv_fp8_cast` (MSE-static weight scales via FP8 scale sweep; dynamic NVFP4 input scales; FP8 KV cache).
- **Group size:** 16. **KV cache:** FP8.
- **Quantized modules:** 69,120 = 60 layers × 384 experts × 3 projections (`gate_proj`, `up_proj`, `down_proj`).
- The `exclude_modules` / `quantization_config` match `nvidia/Kimi-K2.6-NVFP4`.

## Calibration Dataset:
[cnn_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail) + [Nemotron-Post-Training-Dataset-v2](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2); `calib_size=512`, `calib_seq=512`. Full expert coverage was achieved during calibration (0 experts required the max-based amax backstop).

## Usage

To serve this checkpoint with [vLLM](https://github.com/vllm-project/vllm) (`vllm/vllm-openai:latest`):

```sh
VLLM_USE_FLASHINFER_MOE_FP4=1 python3 -m vllm.entrypoints.openai.api_server \
  --model decart-ai/Kimi-K2.7-Code-NVFP4 \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --tool-call-parser kimi_k2 --enable-auto-tool-choice \
  --trust-remote-code
```

Validated load + generation on NVIDIA B300 with vLLM 0.23.

## Reproducibility Notes
- Quantized with NVIDIA Model Optimizer (commit `cc17f2c`).
- Loading the base INT4 checkpoint requires `compressed-tensors==0.14.0.x` (`CompressedLinear` was deprecated in 0.15+).
- Export was run with `CUDA_LAUNCH_BLOCKING=1` to avoid an asynchronous CUDA fault in the decompress/quantize path observed on Blackwell.

## Evaluation
Coding-focused evaluation against the INT4 baseline is pending and will be added.

## Model Limitations
The base model was trained on internet data that may contain toxic language and societal biases; the model may reflect these and may generate inaccurate, incomplete, or otherwise undesirable output. Validate with use-case-specific testing before deployment.