Instructions to use beyoru/KAT-Coder-V2.5-Dev-VL-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use beyoru/KAT-Coder-V2.5-Dev-VL-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="beyoru/KAT-Coder-V2.5-Dev-VL-Flash")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("beyoru/KAT-Coder-V2.5-Dev-VL-Flash")
model = AutoModelForMultimodalLM.from_pretrained("beyoru/KAT-Coder-V2.5-Dev-VL-Flash", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use beyoru/KAT-Coder-V2.5-Dev-VL-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "beyoru/KAT-Coder-V2.5-Dev-VL-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/KAT-Coder-V2.5-Dev-VL-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/beyoru/KAT-Coder-V2.5-Dev-VL-Flash

SGLang

How to use beyoru/KAT-Coder-V2.5-Dev-VL-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "beyoru/KAT-Coder-V2.5-Dev-VL-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/KAT-Coder-V2.5-Dev-VL-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "beyoru/KAT-Coder-V2.5-Dev-VL-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "beyoru/KAT-Coder-V2.5-Dev-VL-Flash",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use beyoru/KAT-Coder-V2.5-Dev-VL-Flash with Docker Model Runner:
```
docker model run hf.co/beyoru/KAT-Coder-V2.5-Dev-VL-Flash
```

KAT-Coder-V2.5-Dev-VL-Flash / README.md

beyoru

Add README.md: sample serving config

d26364c verified 4 days ago

preview code

Raw

History Blame Contribute Delete

7.36 kB

	---
	license: apache-2.0
	base_model:
	- beyoru/KAT-Coder-V2.5-Dev-VL
	pipeline_tag: image-text-to-text
	library_name: transformers
	tags:
	- code
	- agent
	- agentic-coding
	- moe
	- vision-language
	- nvfp4
	- fp4
	- quantized
	- blackwell
	- vllm
	language:
	- en
	- zh
	---

	# KAT-Coder-V2.5-Dev-VL-Flash

	NVFP4 build of [`beyoru/KAT-Coder-V2.5-Dev-VL`](https://huggingface.co/beyoru/KAT-Coder-V2.5-Dev-VL),
	which is itself the vision-enabled build of
	[`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev).

	> ## ⚠️ Requires an NVIDIA Blackwell GPU
	>
	> NVFP4 runs 4-bit weights and 4-bit activations on Blackwell FP4 tensor cores. This checkpoint
	> needs compute capability sm100 / sm120: B200, B300, RTX PRO 6000 Blackwell, RTX 50-series,
	> DGX Spark. It will not run on Hopper (H100/H200), Ada (L40S/RTX 40), or Ampere (A100).
	> On those GPUs use the bf16 build
	> [`beyoru/KAT-Coder-V2.5-Dev-VL`](https://huggingface.co/beyoru/KAT-Coder-V2.5-Dev-VL) instead.

	> ## ⚠️ Required flag: `--max-num-batched-tokens 8192`
	>
	> This is a hybrid attention + Gated-DeltaNet architecture. With prefix caching on (the default in
	> recent vLLM), vLLM forces `mamba_cache_mode='align'` and then raises the attention block size to
	> 2096 so that "attention page size >= mamba page size". The default `max_num_batched_tokens` is
	> 2048, so engine startup dies with:
	>
	> ```
	> AssertionError: In Mamba cache align mode,
	> block_size (2096) must be <= max_num_batched_tokens (2048)
	> ```
	>
	> Any value >= 2096 fixes it; 8192 also helps prefill. (Disabling prefix caching works too — mamba
	> cache falls back to mode `none` — but you lose the prefix cache, which is not worth it.)
	> This applies to the bf16 parent as well.

	## Loading status on Blackwell

	Confirmed loading and initialising on sm121 with vLLM (nightly `0.1.dev18498`):

	```
	Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
	Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend
	Loading weights took 114.37 s -> 20.61 GiB
	FLASH_ATTN for vit attention / MMEncoderAttention <- vision tower initialises
	torch.compile 36.83 s · CUDA graphs PIECEWISE=51 FULL=35 · KV cache 83.33 GiB
	```

	Weights, NVFP4 kernel selection, the MoE backend, the vision tower, compilation and CUDA-graph
	capture all come up cleanly. Generation quality on FP4 tensor cores has still not been benchmarked
	— no accuracy numbers are claimed here. Reports in the Community tab are welcome.

	---

	## Precision layout

	Mixed precision — not everything is FP4. Weight-only 4-bit (W4A16) does not touch FP4 tensor cores,
	so the parts that dominate decode are W4A4, while the parts that hurt accuracy most stay higher.

	\| Component \| Precision \|
	\|---\|---\|
	\| MoE experts, all 40 layers (`mlp.experts..{gate,up,down}_proj`) \| NVFP4 W4A4* (group 16, FP8-E4M3 scales) \|
	\| Shared expert (`shared_expert.{gate,up,down}_proj`) \| NVFP4 W4A4 \|
	\| Attention (`self_attn.{q,k,v,o}_proj`, `linear_attn.{in_proj_qkv,in_proj_z,out_proj}`) \| FP8 W8A8 \|
	\| `lm_head` \| FP8 W8A8 \|
	\| Router (`mlp.gate`, `shared_expert_gate`), `linear_attn.{in_proj_a,in_proj_b,norm}` \| BF16 \|
	\| Vision tower (all `model.visual.`) \| BF16* \|
	\| KV cache \| FP8 (calibrated, per-tensor static) \|

	Pushing every expert layer to W4A4 is the "Fast" trade: maximum FP4 tensor-core coverage, at some
	accuracy cost versus keeping the last layers in FP8.

	\| \| bf16 build \| this build \|
	\|---\|---\|---\|
	\| Size on disk \| 70.2 GB \| see Files and versions \|
	\| Minimum GPU \| 1 × 141 GB, or 2 × 80 GB \| fits comfortably on a single Blackwell card \|

	---

	## Ready-made config

	[`docker-compose.vllm.yaml`](./docker-compose.vllm.yaml) — vLLM on a single Blackwell GPU, vision
	enabled, MoE backend left to auto-selection. Untested on real FP4 hardware (see the warning above).

	SGLang users: FlashQLA's GDN prefill kernel applies to this architecture (30 of 40 layers are
	`linear_attention`) and is opt-in via `--linear-attn-prefill-backend flashqla`. **Windowed-MTP does
	not apply** — `mtp_num_hidden_layers = 0`, so there is no MTP/NEXTN head to self-speculate with.

	## Serving

	### vLLM

	```bash
	uv pip install vllm --torch-backend=auto # vllm >= 0.26.0

	vllm serve beyoru/KAT-Coder-V2.5-Dev-VL-Flash \
	--served-model-name kat-vl-flash \
	--max-model-len 32768 \
	--gpu-memory-utilization 0.90 \
	--trust-remote-code
	```

	Let vLLM pick the MoE backend. Do not pin one. On Blackwell the CUTLASS / FlashInfer-TRTLLM /
	CuTe-DSL NVFP4 paths are the fast ones; forcing Marlin gives a large throughput regression because
	Marlin is a weight-only (W4A16) kernel and never reaches the FP4 tensor cores.

	Do not pass `--language-model-only` — that flag belongs to the upstream text-only release and
	would disable the vision tower this build contains.

	### Transformers

	Loading works (the quantization format is `compressed-tensors`), but transformers has no fast FP4
	execution path — it dequantizes. Use vLLM or SGLang for anything performance-sensitive.

	---

	## Usage

	OpenAI-compatible API. This is a thinking model: it emits a reasoning block ending in `</think>`,
	so pass a generous `max_tokens` (≥256) and strip everything up to `</think>`, or disable thinking via
	`chat_template_kwargs: {"enable_thinking": false}`.

	```python
	from openai import OpenAI
	client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

	r = client.chat.completions.create(
	model="kat-vl-flash",
	messages=[{"role": "user", "content": "Write a Python function to reverse a linked list."}],
	max_tokens=512, temperature=0.0,
	)
	print(r.choices[0].message.content.split("</think>")[-1].strip())
	```

	Sampling defaults follow upstream: `temperature=1.0, top_p=0.95, top_k=20` for open-ended work,
	`temperature=0.0` for deterministic coding.

	---

	## Vision capability — READ THIS

	The vision tower was never co-trained with this checkpoint's language model, which was heavily
	RL-tuned for agentic coding. Measured on the bf16 parent (this build inherits the behaviour, and
	4-bit experts can only make it worse):

	\| probe \| bf16 parent \|
	\|---\|---\|
	\| Dominant colour (4 solid-colour images) \| 4 / 4 \|
	\| Shape (circle / square / triangle) \| 3 / 3 \|
	\| Text/number reading (OCR) \| 0 / 4 — `CAT`→`CCT`, `42`→`48`, `HELLO`→`Hiro` \|

	- ✅ Coarse visual questions: colour, shape, layout, rough scene gist.
	- ❌ Not usable for: reading code from screenshots, OCR, UI labels, document or chart
	understanding — anything where one wrong character changes the answer.

	If your workload is text-only, the vision tower costs you 0.9 GB and nothing else; the language model
	is what this checkpoint is for.

	---

	## Acknowledgements

	- [`Kwaipilot/KAT-Coder-V2.5-Dev`](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) — base model
	and all language-model weights (Apache-2.0).
	- [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) — the architecture this model
	derives from (Apache-2.0).
	- The NVFP4 mixed-precision layout follows the approach popularised by
	[Unsloth's Dynamic NVFP4 quants](https://docs.unsloth.ai/basics/nvfp4): keep the accuracy-critical
	layers at FP8/BF16, push the rest to W4A4 so Blackwell's FP4 tensor cores are actually used.

	Released under Apache-2.0, matching both upstream models.