Instructions to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16")
model = AutoModelForMultimodalLM.from_pretrained("Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

SGLang

How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16
```

KAT-Coder-V2.5-Dev REAP-50 NVFP4A16 (16 GB)

REAP expert-pruned (50%) + NVFP4A16 quantized build of Kwaipilot/KAT-Coder-V2.5-Dev (69.40 SWE-bench Verified claimed), sized and served to run as a local agentic coding model inside 16 GB of consumer VRAM — 12.45 GiB, RTX 5070 Ti (SM120), vLLM. Built with a router-renormalization fix for this architecture (contributed upstream) and a vision tower stripped of its untrained weights.

SWE-bench Verified: 20/50 = 40.0% resolved, via mini-swe-agent's official bash-only scaffold — below the 56.4% bar set by Devstral Small (2512) under the same scaffold. 27 of 50 instances produced no usable patch; 18 of those hit the 32K context ceiling this card's VRAM budget imposes, and the run must be read as context/step-limited, not as an unconditional capability measurement. See "SWE-bench Verified" below before citing the headline number without that context.

Highlights

Result	Detail
12.45 GiB	REAP 50% expert pruning + NVFP4A16 (weight-only, data-free), vision tower stripped
149.5 tok/s median, n=5	Benchmark config: 512 in / 256 out, batch 1, CUDA graphs (FULL_AND_PIECEWISE), 14,672-token context ceiling
89.0% / 90.5%	HumanEval+ [83.3, 92.9] / MBPP+ [87.1, 93.0], greedy, instruct framing
40.0% (20/50)	SWE-bench Verified, `mini-swe-agent` bash-only — see caveats below
28.9 s load	CUDA graphs enabled, no CPU offload

Why 50 percent

Forced by arithmetic on a 16 GB card, not a tuning choice:

variant	size	fits 16 GB
bf16 base	69.3 GB	no
NVFP4, unpruned	21.9 GB	no
REAP 25% + NVFP4	~16–17 GB	no — not once KV cache is counted
REAP 50% + NVFP4	12.45 GiB	yes

Supporting evidence: Half the Experts, All the Code pruned Qwen3.6-35B-A3B, this model's own base, at 50% with no statistically detectable loss on its primary code benchmark.

SWE-bench Verified — read before citing the 40.0% figure alone

metric	value
resolved	20/50 = 40.0%
resolved of completed (valid patch produced)	20/22 = 90.9%
ContextWindowExceeded	18 (32K ceiling)
LimitsExceeded	9
garbage/invalid patch	1 of 23 generated

The scaffold is capped at a 32K-token context window — the safe ceiling this card's VRAM budget supports, not a property of the model. Devstral Small averages 86.9 LM calls/instance under the same scaffold; many KAT-Coder instances hit the context limit before finishing. 90.9% of instances where the model actually produced a patch had that patch resolve the issue — most of the gap to the 56.4% competitive bar is instances that never got to submit a patch at all, not patches that were wrong. This is disclosed as a real result, not an excuse: the 40.0% headline number is the correct number to cite; the breakdown above is the correct context for interpreting it.

Prior art and scope of claims

Verified against the Hugging Face Hub on 2026-08-17:

REAP combined with NVFP4 on qwen3_5_moe already exists (rene98c/Qwen3.5-397B-A17B-REAP-28-NVFP4, March 2026, 23.1K downloads).
REAP on this specific model exists as GGUF (gbuzhf/KAT-Coder-V2.5-Dev-REAP-205E-MTP-GGUF).

What is distinct, and all that is claimed: a vLLM-servable KAT-Coder that is genuinely usable in 16 GB, with published SWE-bench Verified, HumanEval+, and MBPP+ numbers and their confidence intervals — none of which the prior art above publishes.

Quantization and pruning details

Field	Value
Base model	`Kwaipilot/KAT-Coder-V2.5-Dev`
Pruning	REAP, expert-level, 50% compression ratio, seed 42
Pruning calibration	`theblackcat102/evol-codealpaca-v1`, 64 batches/category, 2048 max length
Router renormalization	Fixed (was silently disabled by the upstream REAP adapter for this architecture; committed for upstreaming)
Quantization method	compressed-tensors / llm-compressor, `QuantizationModifier` (PTQ)
Quantization scheme	NVFP4A16 — weight-only, data-free, 82 s
Quantization calibration	`evol-codealpaca` (deliberately not the Magicoder set used for evaluation)
Ignored / kept unquantized	`lm_head`, routers, shared expert gates, embeddings, DeltaNet conv1d + linear-attention projections, MTP module
Vision tower	Removed. The base model declares one and ships no weights for it; this checkpoint contains neither the declaration nor the 333 untrained tensors (0.83 GiB) transformers would otherwise materialise.
Files	3 safetensors shards of ~5 GiB plus `model.safetensors.index.json`, matching the base model's layout, so an interrupted download resumes at shard granularity
Built on	RTX 5070 Ti, 16 GB VRAM, SM120 (compute capability 12.0)
Serving kernel	vLLM selects `MarlinNvFp4LinearKernel` on this card: 4-bit weights are decoded and the GEMM runs in bf16. NVFP4A16 is weight-only, so no FP4 arithmetic is required. A native FP4 path for SM120 exists via FlashInfer CuTeDSL but is not used here and is unmeasured.

Usage

Requires vLLM with SM120 support (CUDA graphs are correct on this card for this model, despite past reports of SM120 CUDA-graph issues on other architectures) and native tool calling for agentic use:

vllm serve Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16 \
  --served-model-name kat-16gb \
  --max-model-len 32768 --max-num-seqs 8 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --enable-prefix-caching --max-num-batched-tokens 4096 \
  --compilation-config '{"cudagraph_capture_sizes":[1,2],"cudagraph_mode":"PIECEWISE"}' \
  --language-model-only

--language-model-only is required: the model declares a vision tower it has no trained weights for, and without this flag vLLM profiles a 16K-token image budget through it. --enable-prefix-caching is the single largest agentic lever measured on this model — 45x on replayed history (0.21 s vs 30.74 s for a 13,130-token history). Never set --max-model-len near a measured ceiling: available KV cache swings 0.49–1.41 GiB with host desktop VRAM use, and higher values fail intermittently rather than at startup.

Evaluation

HumanEval+ and MBPP+ via lm-eval-harness / EvalPlus, greedy decoding, instruct framing, Wilson confidence intervals:

benchmark	score	95% CI	n
HumanEval+	89.0%	[83.3, 92.9]	164
MBPP+	90.5%	[87.1, 93.0]	378

KAT-Coder-V2.5-Dev publishes no HumanEval/MBPP/EvalPlus numbers, so there is no published upstream figure to compare these against.

Both figures were re-measured on the released checkpoint itself and reproduced inside their intervals: HumanEval+ 90.9% [85.5, 94.4] and MBPP+ 89.9% [86.5, 92.6], same problem counts, same greedy decoding. The table reports the original measurement. The two differences run in opposite directions (+1.9 pp and -0.6 pp), which is greedy-decoding nondeterminism under vLLM's batching rather than a difference in weights. Reproduce with bash scripts/eval/eval_suite.sh.

SWE-bench Verified via the official swebench.harness.run_evaluation harness against mini-swe-agent bash-only rollouts (scaffold: SWE-bench/experiments v1.17.2 configuration) — see the dedicated section above for the full breakdown and required caveats.

Known limitations

32K context window is a hardware-forced ceiling, not a design choice — this card's KV-cache budget cannot safely support more. SWE-bench results must be read as context/step-limited.
No pruning-ablation baseline measured. The unpruned model is 69.3 GB bf16 and does not fit this hardware; the accuracy cost of pruning itself (independent of quantization) is not isolated here.
SWE-bench Verified no longer accepts leaderboard submissions outside academia — these numbers are self-reported and independently reproducible from the released evaluation scripts, not a leaderboard entry.

License

Apache 2.0, inherited from the base model Kwaipilot/KAT-Coder-V2.5-Dev and matching the reap and llm-compressor toolchains used to build this checkpoint.

Citation

This checkpoint is derived from Kwaipilot/KAT-Coder-V2.5-Dev. If you use it, please cite the upstream technical report:

@misc{katcoder_v25_2026,
  title={{KAT-Coder-V2.5 Technical Report}},
  author={{KwaiKAT Team}},
  year={2026},
  month={July},
  eprint={2607.05471},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/pdf/2607.05471}
}

Downloads last month: 74

Safetensors

Model size

19B params

Tensor type

BF16

Model tree for Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

Base model

Kwaipilot/KAT-Coder-V2.5-Dev

Quantized

(63)

this model

Dataset used to train Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

Paper for Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16

KAT-Coder-V2.5 Technical Report

Paper • 2607.05471 • Published Jul 6 • 12