Ninnix96

/

KAT-Coder-V2.5-Dev-gguf

like 0

Hi, there. Just a quick heads up:

$ llama-server -hf Ninnix96/KAT-Coder-V2.5-Dev-gguf
0.00.394.961 E common_download_get_hf_plan: no GGUF files found in repository Ninnix96/KAT-Coder-V2.5-Dev-gguf
0.00.394.963 I Available GGUF files:
0.00.394.963 I  - KAT-Coder-V2.5-Dev-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf
0.00.394.980 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.482.422 W srv  llama_server: -----------------
0.00.482.424 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.482.424 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.482.424 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.482.425 W srv  llama_server: -----------------
0.00.482.472 E srv         start: couldn't bind HTTP server socket, hostname: 127.0.0.1, port: 8080
0.00.482.472 I srv    operator(): operator(): cleaning up before exit...
0.00.482.572 E srv  llama_server: exiting due to HTTP server error

the same error happens if I try to add the model to my llama-server "models.ini" (router mode) using the model tag [Ninnix96/KAT-Coder-V2.5-Dev-gguf:IMATRIX]. For some reason HF does not recognize the GGUF, as it is not listed in the model's card as well:

Besides that, some quick results from quick nondeterministic tests:

1. Good performance when compared with APEX I-Compact for basic math/logic prompts
2. It may enter a CoT loop inside the "thinking" tags, probably presence-penalty=1.5 as suggested in the original model card will help
3. I thought this would be much faster than base APEX-I-Compact or IQ4_XS, but it was the same speed or maybe a bit slower, my bet is that some layers are in still in F32 (I use an Ampere card, and I see an improvement in tg speed when using Q8 or at least BF16) (nvm, I checked the other weights that have higher tg and they also use F32 in some layers, so the difference is something else)
4. For some reason, this is the only KAT-Coder-v2.5-Dev variant so far that still "tries to find" an answer for difficult tests, like base Qwen3.6, all the others seem super over-confident and stick to the first option that is generated and go ahead giving it as (wrong) answer

Have you tried a maybe not so extreme scheme for quantization, maybe Q3_K/IQ3_XSS? Is there a big difference in size? My bet is that reducing the remaining F32 layers to Q8_0 will improve tg speed while keeping the quality, just a thought...

Thank you for providing the weights and detailed info in the model card. Regards