Instructions to use Myric/KAT-Coder-V2.5-Dev-MTP-head with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Myric/KAT-Coder-V2.5-Dev-MTP-head
# Run inference directly in the terminal:
llama cli -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Myric/KAT-Coder-V2.5-Dev-MTP-head
# Run inference directly in the terminal:
llama cli -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Myric/KAT-Coder-V2.5-Dev-MTP-head
# Run inference directly in the terminal:
./llama-cli -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Myric/KAT-Coder-V2.5-Dev-MTP-head
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Use Docker

docker model run hf.co/Myric/KAT-Coder-V2.5-Dev-MTP-head

LM Studio
Jan
Ollama
How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Ollama:
```
ollama run hf.co/Myric/KAT-Coder-V2.5-Dev-MTP-head
```

Unsloth Studio

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Myric/KAT-Coder-V2.5-Dev-MTP-head to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Myric/KAT-Coder-V2.5-Dev-MTP-head to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Myric/KAT-Coder-V2.5-Dev-MTP-head to start chatting

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Myric/KAT-Coder-V2.5-Dev-MTP-head"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Myric/KAT-Coder-V2.5-Dev-MTP-head" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Docker Model Runner:
```
docker model run hf.co/Myric/KAT-Coder-V2.5-Dev-MTP-head
```

Lemonade

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Myric/KAT-Coder-V2.5-Dev-MTP-head

Run and chat with the model

lemonade run user.KAT-Coder-V2.5-Dev-MTP-head-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use Myric/KAT-Coder-V2.5-Dev-MTP-head with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Myric/KAT-Coder-V2.5-Dev-MTP-head

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Myric/KAT-Coder-V2.5-Dev-MTP-head

Run Hermes

hermes

Atomic Chat

KAT-Coder-V2.5-Dev-MTP-head / README.md

Myric

Upload README.md with huggingface_hub

2f91c61 verified 2 days ago

preview code

Raw

History Blame Contribute Delete

5.26 kB

	---
	license: apache-2.0
	base_model: Kwaipilot/KAT-Coder-V2.5-Dev
	tags:
	- mtp
	- speculative-decoding
	- qwen3_5_moe
	- experiment
	---

	# KAT-Coder-V2.5-Dev — MTP head (experimental, un-fine-tuned baseline)

	[Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) ships with
	`mtp_num_hidden_layers: 0` — no MTP/NextN head at all, so no speculative-decoding speedup is
	possible out of the box. This repo adds one back by transplanting
	[Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)'s own trained MTP head onto
	KAT-Coder. This is architecturally clean, not a hack: every relevant dimension (hidden_size,
	expert count, attention/MoE shapes) matches exactly between the two models, since KAT-Coder is
	itself built on the Qwen3.6-35B-A3B backbone.

	This is the naive, un-fine-tuned transplant — the MTP head's weights are copied directly from
	stock Qwen3.6 with zero training on KAT-Coder's own hidden-state distribution. A fine-tuning pass
	(freezing KAT-Coder's own 40 layers entirely, training only this ~850M-parameter head) is a
	planned follow-up; this repo documents the honest baseline before that.

	## What's actually in this repo

	Just one new shard (`model-00014-of-mtp.safetensors`, 1.69GB) containing the 19 MTP-related
	tensors, plus a `config.json`/`model.safetensors.index.json` already merged to reference it
	alongside KAT-Coder's own 13 original shards. Nothing else was touched — KAT-Coder's own 13
	shards are byte-identical to the original release; this is a drop-in addition, not a full
	re-upload of the 65GB checkpoint.

	To use: put this repo's 3 files into a local copy of
	[Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) alongside its
	own 13 `model-*-of-00013.safetensors` shards (the `config.json`/index here already reference all
	14 shards together).

	## Baseline: draft-acceptance rate (before fine-tuning)

	Measured via `llama-server --spec-type draft-mtp --spec-draft-n-max 2` on an APEX i-quality quant
	(4.67 bpw) built from this checkpoint, 5 real from-scratch coding prompts, `n_predict=400` each:

	\| prompt \| draft tokens \| accepted \| rate \| tok/s \|
	\|---\|---:\|---:\|---:\|---:\|
	\| binary search tree (insert/delete/traverse) \| 286 \| 255 \| 89.2% \| 105.5 \|
	\| CSV parser (stdlib only) \| 114 \| 84 \| 73.7% \| 94.6 \|
	\| retry decorator w/ exponential backoff \| 188 \| 152 \| 80.9% \| 100.6 \|
	\| merge two sorted linked lists \| 74 \| 58 \| 78.4% \| 97.6 \|
	\| thread-safe LRU cache \| 360 \| 218 \| 60.6% \| 85.9 \|
	\| overall \| 1022 \| 767 \| 75.1% \| — \|

	For reference, the same KAT-Coder quant tier without any MTP head runs at ~70.8 tok/s solo
	decode speed (measured separately, no speculative decoding available at all). So even this
	un-fine-tuned transplant is already a real ~20-40% speedup, not a wash — acceptance rate varies
	by task in a way that tracks predictability (the BST task, mostly standard tree-traversal
	boilerplate, hits 89%; the LRU cache, with genuine locking/eviction logic, is the hardest at 61%).

	## Full coding-benchmark validation (real agentic tasks, not just completion speed)

	Ran the same two task suites used elsewhere in this project's Qwen3.6-lineage work (an existing
	5-task hard tier, plus a new 3-task tier designed to require genuine algorithmic reasoning rather
	than spec-transcription) against this MTP-enabled build vs the same quant without MTP, via
	`opencode`'s real agentic tool-calling loop (not just raw completion). **Correctness was
	identical — 100% on both suites, both with and without MTP** — exactly what the theory predicts:
	the MTP head only drafts candidate tokens, the main model always verifies against its own true
	computation, so a bad draft just gets rejected and never reaches the output. Speed differed
	substantially:

	\| hard-tier task \| no MTP \| with MTP \| speedup \|
	\|---\|---:\|---:\|---:\|
	\| batch_scheduler \| 22.5s \| 20.8s \| 1.08x \|
	\| buddy_allocator \| 123.5s \| 63.5s \| 1.94x \|
	\| circuit_breaker \| 50.7s \| 14.0s \| 3.62x \|
	\| expr_parser_advanced \| 54.7s \| 21.1s \| 2.59x \|
	\| inventory_fix_and_extend \| 16.4s \| 12.3s \| 1.33x \|
	\| total \| 267.8s \| 131.7s \| 2.03x \|

	\| tier-4 task \| no MTP \| with MTP \| speedup \|
	\|---\|---:\|---:\|---:\|
	\| lru_ttl_cache \| 20.5s \| 25.2s \| 0.81x \|
	\| rate_limiter_fix_and_extend \| 19.7s \| 12.1s \| 1.63x \|
	\| weighted_interval_scheduling \| 22.5s \| 17.4s \| 1.29x \|
	\| total \| 62.7s \| 54.7s \| 1.15x \|

	Both suites hit 100% (59/59 and 28/28 test cases respectively) in both configurations. The one
	slower result (`lru_ttl_cache`, 0.81x) is most likely ordinary run-to-run variance in how much
	iteration that specific attempt needed, not a systematic MTP cost — every other task improved, and
	the hard-tier set as a whole is a genuine 2x.

	## Attribution

	- KAT-Coder base model: Kwaipilot — [Kwaipilot/KAT-Coder-V2.5-Dev](https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev) (Apache-2.0).
	- MTP head weights: Qwen — [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Apache-2.0), transplanted unmodified.
	- Quantization/serving: llama.cpp ([ggml-org](https://github.com/ggml-org/llama.cpp)).

	Unofficial community experiment; not affiliated with or endorsed by Kwaipilot or Qwen.