Kwaipilot

/

KAT-Coder-V2.5-Dev

like 316

Follow

Kwaipilot 587

Thanks for raising these thoughtful questions!
A primary factor is that it can be challenging to exactly reproduce the official reported benchmark numbers. Notably, the Qwen team has targeted optimizations on certain evaluation datasets. To ensure fair comparisons across all models, we re-run all evaluations under a unified, consistent standard setup. You can find detailed evaluation settings described in the footnote text below our benchmark table.

I actually tested these two models. Previously, I used Ornith, and I felt that KAT-Coder was a bit better—just my personal impression, without any benchmarking.

I agree, I have tested Ornith, Agents-a1, Ternary Bonsai, Laguna XS2.1 but this one is not in the same category.
And I’m not talking about standard benchmarks; I’m talking about a real software development environment with over 100 files and 50,000 lines of code.
I feel like we are very very close to Qwen3.6 27B...
Happy to get this on my old RTX3060 with 260k context window.
Very good work !

First of all, thank you for responding @liushiyang . I've checked and you ran qwen3.6-35b-a3b at SWE-bench Verified / Multilingual / Pro, KAT-Code-Bench with these parameters: pass@k=1, temperature=1.0, top_p=0.95, 256k ctx.

While Qwen recommends these to be used for coding tasks:

• "Thinking mode for precise coding tasks (e.g. WebDev): temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0"

Qwen did state they used the following parameters on their benchmarks:

• "SWE-Bench Series: Internal agent scaffold (bash + file-edit tools); temp=1.0, top_p=0.95, 200K context window."
• Though this line might indicate some cherry picking in their eval: "We correct some problematic tasks in the public set of SWE-bench Pro and evaluate all baselines on the refined benchmark."

Deepreinforce-ai's recommends the same parameters as Qwen's for coding tasks but their reported parameters used on their own benchmarks were:

• "Terminal-Bench 2.1 (Claude Code): We evaluate Terminal-Bench 2.1 using Claude Code 2.1.126 with parser=json, temperature=1.0, top_p=1.0, max_new_tokens=131072. Results are averaged over 5 runs. Again, Qwen chat template needs to be modified."
• "SWE-Bench Verified, Pro and Multilingual: using OpenHands harness with temp=1.0, top_p=0.95, 256k context window."

Not sure how your model server behaves if some of these are not set explicitly.

One question I would think of as interesting right now would be:
Why didn't you do multiple runs per model on the benchmarks?

@space-spike
Thanks for the thoughtful question!

There are two main reasons we didn’t run multiple repeated evaluations for each checkpoint:
First, we have a large pool of candidate checkpoints to evaluate. Running 5 repeated trials for every candidate would lead to an infeasible computational overhead.
Second, if a single model achieves strong results consistently across all 7 benchmarks, we argue that high scores across such a diverse set of tasks are unlikely to come purely from random sampling luck, making repeated runs less necessary for filtering promising candidates.

Understood. Thanks for the transparency and the engagement from the Kwaipilot team.

I'll be testing the model myself in day-to-day coding and my local evals, so I'll maybe have more feedback and food for thought. Cheers 😄

Ran my evals (math and coding problems) with the q4_k_m.gguf quant by bartowski. Though my quick benchmarks are not hard enough for this model class, the Kat-Coder-v2.5 completes the problems with a fraction of thinking tokens used compared to vanilla qwen3.6-35b-a3b and ornith-1.0-35b. The traces contain genuine reasoning, and the model doesn't ramble almost at all, it thinks clearly and follows through the problem steps efficiently. If that causes it to catch less of its mistakes on longer harder problems is yet to be seen.

I'm impressed with this models reasoning, what thinking traces did you use in the RL part of training?
Any plans for an MTP/Dflash/DSpark head for this model?
Tried using mtp and dflash from qwen3.6-35b-a3b as spec-draft for this model, they work alright.
If I might, I suggest you take a look at unlsoth's chat template for the qwen3.6-35b-a3b. It's mostly the same as yours but a bit more robust for coding agent harnesses and it could work well for kat-coder-v2.5-dev as well.

For me it solved C coding task much better and faster than 3.6 35b and even 27b but that was since it is moe in the end it is faster bat would say same quality.

@vlarmet
@space-spike
@Joske0000
Guys thank you for your feedback. Please share your config you've used to run it with llama.cpp. KV cache quantization keys, etc.

llama-server.exe -m Kwaipilot_KAT-Coder-V2.5-Dev-Q5_K_M.gguf 
--no-mmap 
-t 4 
--cache-reuse 256  
--jinja 
--chat-template-file D:\llama.cpp\chat_template.jinja 
--reasoning-format deepseek 
--reasoning-budget 8000   
--temp 0.7 
--top-p 0.8 
--top-k 20 
--presence-penalty 1.5 
-c 260000  
-ngl 99 
--n-cpu-moe 35 
-fa on 
--host 0.0.0.0 
--port 8080

note : I use Froggeric chat template and -ctk and -ctv are f16 by default