Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4-W4A4

KAT-Coder-V2.5-Dev REAP-50 W4A4

This is the same REAP-50-pruned base as Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4A16, quantized with a different scheme: activations quantized to 4 bits as well as weights (NVFP4, i.e. W4A4), instead of the sibling release's weight-only NVFP4A16. Same source checkpoint, same ignore list — only the quantization scheme differs, so the two are a clean, direct comparison.

Both builds are legitimate choices. This card documents the measured trade-offs so you can pick based on what matters for your use case, and publishes the raw comparison so the results are independently checkable.

Why two builds

Weight-only quantization (NVFP4A16) dequantizes on the fly and computes in bf16 — there is no hardware path for a weight-only scheme to reach a GPU's native FP4×FP4 tensor-core instructions, on any GPU. Quantizing activations too (this build) does reach those native kernels (FlashInferCutlassNvFp4LinearKernel for dense layers, FLASHINFER_CUTLASS for MoE experts, confirmed via vLLM's own kernel-selection logging). The question this build answers: does native FP4×FP4 compute actually beat a dequant-then-bf16-compute fallback on consumer Blackwell (SM120, e.g. RTX 5070 Ti/5080/5090) for a real agentic-coding decode workload?

Measurements

5 interleaved process invocations per arm (warmup discarded), median + range reported, under both isolated eager-mode execution (cleanly isolates kernel dispatch, but understates real throughput by ~7x on this hardware) and the same PIECEWISE CUDA-graph configuration the NVFP4A16 model serves with in production:

	HumanEval	HumanEval+	MBPP+	decode, eager	decode, PIECEWISE (production)
NVFP4A16 (sibling release)	95.7%	90.9%	89.9%	18.8 tok/s	142.5 tok/s
W4A4 (this build)	92.07%	89.02%	91.01%	14.5 tok/s	119.2 tok/s

On this hardware, for this single-stream (batch=1) workload: NVFP4A16 is faster (0.84x for W4A4 under the production PIECEWISE configuration, 0.77x in isolated eager mode). Both numbers are fast in absolute terms — 119 tok/s is roughly 17-20x typical human reading speed and comfortably interactive for coding use; the gap is relative to the sibling build, not a usability threshold. Accuracy is mixed rather than uniformly different: W4A4 is lower on HumanEval and HumanEval+, and very slightly higher on MBPP+ (344/378 vs 340/378 — one problem's difference on a 378-problem set). For reference, QSpec-era literature (INT4-generation quantization) reported W4A4 losing up to 38.73% on HumanEval; that collapse did not reproduce here, consistent with NVFP4's per-16-block scaling and FP8 scale factors being a better-conditioned format than INT4-era quantization.

Same checkpoint size either way (12.4532 GiB here vs 12.4512 GiB for A16).

Which build to use

• NVFP4A16 (the sibling release) is faster in our measurements and is the one we default to for our own agentic-coding pipeline.
• This W4A4 build is the one to reach for if you specifically want the native FP4×FP4 tensor-core execution path (e.g. building on top of activation quantization, or targeting a serving stack where that path matters more than it did for us), or if you want to independently verify or extend the comparison above.

We only measured single-stream (batch=1) decode, matching our own agentic use case. We have not measured batch sizes above 1, where the native kernel's throughput/latency characteristics may differ from the dequant path's — if you test that, we'd like to hear what you find.

Mechanism notes

Two findings surfaced while measuring this, independent of the headline numbers:

• At identical gpu_memory_utilization, the NVFP4A16/Marlin arm needed a higher utilization setting to reliably allocate KV cache than this W4A4 build did at the same checkpoint size — Marlin's dequantize-on-the-fly path appears to need more non-weight runtime workspace than the native FP4 kernel path does.
• W4A4 gains proportionally more from CUDA graphs than NVFP4A16 does (eager→PIECEWISE: NVFP4A16 7.6x, W4A4 8.2x), which narrows the gap between them (0.77x → 0.84x) without closing it.

Usage

Loads and serves like the NVFP4A16 sibling (same architecture, transformers/vllm requirements, SM120/compute-capability-12.0 requirement, no CPU offload needed at 12.45 GiB). See the sibling release's model card for full serving instructions, environment requirements, and the REAP-pruning background — this card documents only what differs about this build.

from vllm import LLM, SamplingParams

llm = LLM(
    model="Ttimms/KAT-Coder-V2.5-Dev-REAP-50-NVFP4-W4A4",
    dtype="bfloat16",
    language_model_only=True,
    trust_remote_code=True,
)

License

Apache 2.0, inherited from the base model Kwaipilot/KAT-Coder-V2.5-Dev and its upstream lineage.