Instructions to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16")
model = AutoModelForMultimodalLM.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

SGLang

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16
```

KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 / README.md

Ttimms

Document shard layout, serving kernel, and add fp4/4-bit tags

0c556f6 verified about 21 hours ago

preview code

Raw

History Blame Contribute Delete

9.18 kB

	---
	base_model:
	- Kwaipilot/KAT-Coder-V2.5-Dev
	base_model_relation: quantized
	license: apache-2.0
	library_name: transformers
	datasets:
	- theblackcat102/evol-codealpaca-v1
	tags:
	- reap
	- pruning
	- nvfp4
	- nvfp4a16
	- fp4
	- 4-bit
	- compressed-tensors
	- quantization
	- vllm
	- blackwell
	- moe
	- agentic-coding
	- swe-bench
	pipeline_tag: text-generation
	---

	# KAT-Coder-V2.5-Dev REAP-50 NVFP4A16 (16 GB)

	**REAP expert-pruned (50%) + NVFP4A16 quantized build of `Kwaipilot/KAT-Coder-V2.5-Dev`
	(69.40 SWE-bench Verified claimed), sized and served to run as a local agentic
	coding model inside 16 GB of consumer VRAM** — 12.45 GiB, RTX 5070 Ti (SM120),
	vLLM. Built with a router-renormalization fix for this architecture (contributed
	upstream) and a vision tower stripped of its untrained weights.

	SWE-bench Verified: 20/50 = 40.0% resolved, via `mini-swe-agent`'s official
	bash-only scaffold — below the 56.4% bar set by Devstral Small (2512) under the
	same scaffold. 27 of 50 instances produced no usable patch; 18 of those hit the
	32K context ceiling this card's VRAM budget imposes, and the run must be read
	as context/step-limited, not as an unconditional capability measurement. See
	"SWE-bench Verified" below before citing the headline number without that
	context.

	## Highlights

	\| Result \| Detail \|
	\|---\|---\|
	\| 12.45 GiB \| REAP 50% expert pruning + NVFP4A16 (weight-only, data-free), vision tower stripped \|
	\| 149.5 tok/s median, n=5 \| Benchmark config: 512 in / 256 out, batch 1, CUDA graphs (FULL_AND_PIECEWISE), 14,672-token context ceiling \|
	\| 89.0% / 90.5% \| HumanEval+ [83.3, 92.9] / MBPP+ [87.1, 93.0], greedy, instruct framing \|
	\| 40.0% (20/50) \| SWE-bench Verified, `mini-swe-agent` bash-only — see caveats below \|
	\| 28.9 s load \| CUDA graphs enabled, no CPU offload \|

	## Why 50 percent

	Forced by arithmetic on a 16 GB card, not a tuning choice:

	\| variant \| size \| fits 16 GB \|
	\|---\|---:\|:---:\|
	\| bf16 base \| 69.3 GB \| no \|
	\| NVFP4, unpruned \| 21.9 GB \| no \|
	\| REAP 25% + NVFP4 \| ~16–17 GB \| no — not once KV cache is counted \|
	\| REAP 50% + NVFP4 \| 12.45 GiB \| yes \|

	Supporting evidence: [Half the Experts, All the Code](https://arxiv.org/html/2607.16721)
	pruned Qwen3.6-35B-A3B, this model's own base, at 50% with no statistically
	detectable loss on its primary code benchmark.

	## SWE-bench Verified — read before citing the 40.0% figure alone

	\| metric \| value \|
	\|---\|---:\|
	\| resolved \| 20/50 = 40.0% \|
	\| resolved of completed (valid patch produced) \| 20/22 = 90.9% \|
	\| ContextWindowExceeded \| 18 (32K ceiling) \|
	\| LimitsExceeded \| 9 \|
	\| garbage/invalid patch \| 1 of 23 generated \|

	The scaffold is capped at a 32K-token context window — the safe ceiling this
	card's VRAM budget supports, not a property of the model. Devstral Small
	averages 86.9 LM calls/instance under the same scaffold; many KAT-Coder
	instances hit the context limit before finishing. **90.9% of instances where
	the model actually produced a patch had that patch resolve the issue** — most
	of the gap to the 56.4% competitive bar is instances that never got to submit
	a patch at all, not patches that were wrong. This is disclosed as a real
	result, not an excuse: the 40.0% headline number is the correct number to
	cite; the breakdown above is the correct context for interpreting it.

	## Prior art and scope of claims

	Verified against the Hugging Face Hub on 2026-08-17:

	- REAP combined with NVFP4 on `qwen3_5_moe` already exists
	([`rene98c/Qwen3.5-397B-A17B-REAP-28-NVFP4`](https://huggingface.co/rene98c/Qwen3.5-397B-A17B-REAP-28-NVFP4),
	March 2026, 23.1K downloads).
	- REAP on this specific model exists as GGUF
	([`gbuzhf/KAT-Coder-V2.5-Dev-REAP-205E-MTP-GGUF`](https://huggingface.co/gbuzhf/KAT-Coder-V2.5-Dev-REAP-205E-MTP-GGUF)).

	What is distinct, and all that is claimed: a **vLLM-servable KAT-Coder that is
	genuinely usable in 16 GB**, with published SWE-bench Verified, HumanEval+, and
	MBPP+ numbers and their confidence intervals — none of which the prior art
	above publishes.

	## Quantization and pruning details

	\| Field \| Value \|
	\|---\|---\|
	\| Base model \| `Kwaipilot/KAT-Coder-V2.5-Dev` \|
	\| Pruning \| REAP, expert-level, 50% compression ratio, seed 42 \|
	\| Pruning calibration \| `theblackcat102/evol-codealpaca-v1`, 64 batches/category, 2048 max length \|
	\| Router renormalization \| Fixed (was silently disabled by the upstream REAP adapter for this architecture; committed for upstreaming) \|
	\| Quantization method \| compressed-tensors / llm-compressor, `QuantizationModifier` (PTQ) \|
	\| Quantization scheme \| NVFP4A16 — weight-only, data-free, 82 s \|
	\| Quantization calibration \| `evol-codealpaca` (deliberately not the Magicoder set used for evaluation) \|
	\| Ignored / kept unquantized \| `lm_head`, routers, shared expert gates, embeddings, DeltaNet conv1d + linear-attention projections, MTP module \|
	\| Vision tower \| Removed. The base model declares one and ships no weights for it; this checkpoint contains neither the declaration nor the 333 untrained tensors (0.83 GiB) transformers would otherwise materialise. \|
	\| Files \| 3 safetensors shards of ~5 GiB plus `model.safetensors.index.json`, matching the base model's layout, so an interrupted download resumes at shard granularity \|
	\| Built on \| RTX 5070 Ti, 16 GB VRAM, SM120 (compute capability 12.0) \|
	\| Serving kernel \| vLLM selects `MarlinNvFp4LinearKernel` on this card: 4-bit weights are decoded and the GEMM runs in bf16. NVFP4A16 is weight-only, so no FP4 arithmetic is required. A native FP4 path for SM120 exists via FlashInfer CuTeDSL but is not used here and is unmeasured. \|

	## Usage

	Requires vLLM with SM120 support (CUDA graphs are correct on this card for
	this model, despite past reports of SM120 CUDA-graph issues on other
	architectures) and native tool calling for agentic use:

	```bash
	vllm serve Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 \
	--served-model-name kat-16gb \
	--max-model-len 32768 --max-num-seqs 8 \
	--gpu-memory-utilization 0.92 --kv-cache-dtype fp8 \
	--reasoning-parser qwen3 \
	--enable-auto-tool-choice --tool-call-parser qwen3_xml \
	--enable-prefix-caching --max-num-batched-tokens 4096 \
	--compilation-config '{"cudagraph_capture_sizes":[1,2],"cudagraph_mode":"PIECEWISE"}' \
	--language-model-only
	```

	`--language-model-only` is required: the model declares a vision tower it has
	no trained weights for, and without this flag vLLM profiles a 16K-token image
	budget through it. `--enable-prefix-caching` is the single largest agentic
	lever measured on this model — 45x on replayed history (0.21 s vs 30.74 s for
	a 13,130-token history). Never set `--max-model-len` near a measured ceiling:
	available KV cache swings 0.49–1.41 GiB with host desktop VRAM use, and higher
	values fail intermittently rather than at startup.

	## Evaluation

	HumanEval+ and MBPP+ via lm-eval-harness / EvalPlus, greedy decoding, instruct
	framing, Wilson confidence intervals:

	\| benchmark \| score \| 95% CI \| n \|
	\|---\|---:\|---:\|---:\|
	\| HumanEval+ \| 89.0% \| [83.3, 92.9] \| 164 \|
	\| MBPP+ \| 90.5% \| [87.1, 93.0] \| 378 \|

	`KAT-Coder-V2.5-Dev` publishes no HumanEval/MBPP/EvalPlus numbers, so there is
	no published upstream figure to compare these against.

	Both figures were re-measured on the released checkpoint itself and reproduced
	inside their intervals: HumanEval+ 90.9% [85.5, 94.4] and MBPP+ 89.9%
	[86.5, 92.6], same problem counts, same greedy decoding. The table reports the
	original measurement. The two differences run in opposite directions
	(+1.9 pp and -0.6 pp), which is greedy-decoding nondeterminism under vLLM's
	batching rather than a difference in weights. Reproduce with
	`bash scripts/eval/eval_suite.sh`.

	SWE-bench Verified via the official `swebench.harness.run_evaluation` harness
	against `mini-swe-agent` bash-only rollouts (scaffold: SWE-bench/experiments
	v1.17.2 configuration) — see the dedicated section above for the full
	breakdown and required caveats.

	## Known limitations

	- 32K context window is a hardware-forced ceiling, not a design choice —
	this card's KV-cache budget cannot safely support more. SWE-bench results
	must be read as context/step-limited.
	- No pruning-ablation baseline measured. The unpruned model is 69.3 GB
	bf16 and does not fit this hardware; the accuracy cost of pruning itself
	(independent of quantization) is not isolated here.
	- SWE-bench Verified no longer accepts leaderboard submissions outside
	academia — these numbers are self-reported and independently reproducible
	from the released evaluation scripts, not a leaderboard entry.

	## License

	Apache 2.0, inherited from the base model `Kwaipilot/KAT-Coder-V2.5-Dev` and
	matching the `reap` and `llm-compressor` toolchains used to build this checkpoint.

	## Citation

	This checkpoint is derived from `Kwaipilot/KAT-Coder-V2.5-Dev`. If you use it,
	please cite the upstream technical report:

	```bibtex
	@misc{katcoder_v25_2026,
	title={{KAT-Coder-V2.5 Technical Report}},
	author={{KwaiKAT Team}},
	year={2026},
	month={July},
	eprint={2607.05471},
	archivePrefix={arXiv},
	primaryClass={cs.AI},
	url={https://arxiv.org/pdf/2607.05471}
	}
	```