README.md

---
license: apache-2.0
base_model:
- beyoru/KAT-Coder-V2.5-Dev-VL
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- code
- agent
- agentic-coding
- moe
- vision-language
- nvfp4
- fp4
- quantized
- blackwell
- vllm
language:
- en
- zh
---

# KAT-Coder-V2.5-Dev-VL-Flash

**NVFP4 build** of [`beyoru/KAT-Coder-V2.5-Dev-VL`](https://huggingface.co/beyoru/KAT-Coder-V2.5-Dev-VL),
which is itself the vision-enabled build of
[`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev).

> ## ⚠️ Requires an NVIDIA **Blackwell** GPU
>
> NVFP4 runs 4-bit weights **and** 4-bit activations on Blackwell FP4 tensor cores. This checkpoint
> needs compute capability **sm100 / sm120**: B200, B300, RTX PRO 6000 Blackwell, RTX 50-series,
> DGX Spark. It will **not** run on Hopper (H100/H200), Ada (L40S/RTX 40), or Ampere (A100).
> On those GPUs use the bf16 build
> [`beyoru/KAT-Coder-V2.5-Dev-VL`](https://huggingface.co/beyoru/KAT-Coder-V2.5-Dev-VL) instead.

> ## ⚠️ Required flag: `--max-num-batched-tokens 8192`
>
> This is a hybrid attention + Gated-DeltaNet architecture. With prefix caching on (the default in
> recent vLLM), vLLM forces `mamba_cache_mode='align'` and then raises the attention block size to
> **2096** so that "attention page size >= mamba page size". The default `max_num_batched_tokens` is
> **2048**, so engine startup dies with:
>
> ```
> AssertionError: In Mamba cache align mode,
>                 block_size (2096) must be <= max_num_batched_tokens (2048)
> ```
>
> Any value >= 2096 fixes it; 8192 also helps prefill. (Disabling prefix caching works too — mamba
> cache falls back to mode `none` — but you lose the prefix cache, which is not worth it.)
> This applies to the bf16 parent as well.

## Loading status on Blackwell

Confirmed loading and initialising on **sm121** with vLLM (nightly `0.1.dev18498`):

```
Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
Loading weights took 114.37 s -> 20.61 GiB
FLASH_ATTN for vit attention / MMEncoderAttention     <- vision tower initialises
torch.compile 36.83 s · CUDA graphs PIECEWISE=51 FULL=35 · KV cache 83.33 GiB
```

Weights, NVFP4 kernel selection, the MoE backend, the vision tower, compilation and CUDA-graph
capture all come up cleanly. **Generation quality on FP4 tensor cores has still not been benchmarked**
— no accuracy numbers are claimed here. Reports in the Community tab are welcome.

---

## Precision layout

Mixed precision — not everything is FP4. Weight-only 4-bit (W4A16) does not touch FP4 tensor cores,
so the parts that dominate decode are W4**A4**, while the parts that hurt accuracy most stay higher.

| Component | Precision |
|---|---|
| MoE experts, all 40 layers (`mlp.experts.*.{gate,up,down}_proj`) | **NVFP4 W4A4** (group 16, FP8-E4M3 scales) |
| Shared expert (`shared_expert.{gate,up,down}_proj`) | **NVFP4 W4A4** |
| Attention (`self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`) | FP8 W8A8 |
| `lm_head` | FP8 W8A8 |
| Router (`mlp.gate`, `shared_expert_gate`), `linear_attn.{in_proj_a,in_proj_b,norm}` | BF16 |
| **Vision tower** (all `model.visual.*`) | **BF16** |
| KV cache | FP8 (calibrated, per-tensor static) |

Pushing every expert layer to W4A4 is the "Fast" trade: maximum FP4 tensor-core coverage, at some
accuracy cost versus keeping the last layers in FP8.

| | bf16 build | this build |
|---|---|---|
| Size on disk | 70.2 GB | see *Files and versions* |
| Minimum GPU | 1 × 141 GB, or 2 × 80 GB | fits comfortably on a single Blackwell card |

---

## Ready-made config

[`docker-compose.vllm.yaml`](./docker-compose.vllm.yaml) — vLLM on a single Blackwell GPU, vision
enabled, MoE backend left to auto-selection. Untested on real FP4 hardware (see the warning above).

SGLang users: FlashQLA's GDN prefill kernel applies to this architecture (30 of 40 layers are
`linear_attention`) and is opt-in via `--linear-attn-prefill-backend flashqla`. **Windowed-MTP does
not apply** — `mtp_num_hidden_layers = 0`, so there is no MTP/NEXTN head to self-speculate with.

## Serving

### vLLM

```bash
uv pip install vllm --torch-backend=auto      # vllm >= 0.26.0

vllm serve beyoru/KAT-Coder-V2.5-Dev-VL-Flash \
    --served-model-name kat-vl-flash \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code
```

**Let vLLM pick the MoE backend.** Do not pin one. On Blackwell the CUTLASS / FlashInfer-TRTLLM /
CuTe-DSL NVFP4 paths are the fast ones; forcing Marlin gives a large throughput regression because
Marlin is a weight-only (W4A16) kernel and never reaches the FP4 tensor cores.

Do **not** pass `--language-model-only` — that flag belongs to the upstream text-only release and
would disable the vision tower this build contains.

### Transformers

Loading works (the quantization format is `compressed-tensors`), but transformers has no fast FP4
execution path — it dequantizes. Use vLLM or SGLang for anything performance-sensitive.

---

## Usage

OpenAI-compatible API. This is a **thinking model**: it emits a reasoning block ending in `</think>`,
so pass a generous `max_tokens` (≥256) and strip everything up to `</think>`, or disable thinking via
`chat_template_kwargs: {"enable_thinking": false}`.

```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

r = client.chat.completions.create(
    model="kat-vl-flash",
    messages=[{"role": "user", "content": "Write a Python function to reverse a linked list."}],
    max_tokens=512, temperature=0.0,
)
print(r.choices[0].message.content.split("</think>")[-1].strip())
```

Sampling defaults follow upstream: `temperature=1.0, top_p=0.95, top_k=20` for open-ended work,
`temperature=0.0` for deterministic coding.

---

## Vision capability — READ THIS

The vision tower was never co-trained with this checkpoint's language model, which was heavily
RL-tuned for agentic coding. Measured on the **bf16 parent** (this build inherits the behaviour, and
4-bit experts can only make it worse):

| probe | bf16 parent |
|---|---|
| Dominant colour (4 solid-colour images) | 4 / 4 |
| Shape (circle / square / triangle) | 3 / 3 |
| **Text/number reading (OCR)** | **0 / 4** — `CAT`→`CCT`, `42`→`48`, `HELLO`→`Hiro` |

- ✅ Coarse visual questions: colour, shape, layout, rough scene gist.
- ❌ **Not usable for**: reading code from screenshots, OCR, UI labels, document or chart
  understanding — anything where one wrong character changes the answer.

If your workload is text-only, the vision tower costs you 0.9 GB and nothing else; the language model
is what this checkpoint is for.

---

## Acknowledgements

- [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) — base model
  and all language-model weights (Apache-2.0).
- [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — the architecture this model
  derives from (Apache-2.0).
- The NVFP4 mixed-precision layout follows the approach popularised by
  [Unsloth's Dynamic NVFP4 quants](https://docs.unsloth.ai/basics/nvfp4): keep the accuracy-critical
  layers at FP8/BF16, push the rest to W4A4 so Blackwell's FP4 tensor cores are actually used.

Released under Apache-2.0, matching both upstream models.