Instructions to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="pearsonkyle/tmax-27b-imatrix-MTP-GGUF",
	filename="tmax-27b-IQ2_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
# Run inference directly in the terminal:
llama cli -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
# Run inference directly in the terminal:
llama cli -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
# Run inference directly in the terminal:
./llama-cli -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Use Docker

docker model run hf.co/pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

LM Studio
Jan

vLLM

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pearsonkyle/tmax-27b-imatrix-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pearsonkyle/tmax-27b-imatrix-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Ollama
How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Ollama:
```
ollama run hf.co/pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
```

Unsloth Studio

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for pearsonkyle/tmax-27b-imatrix-MTP-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for pearsonkyle/tmax-27b-imatrix-MTP-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for pearsonkyle/tmax-27b-imatrix-MTP-GGUF to start chatting

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
```

Lemonade

How to use pearsonkyle/tmax-27b-imatrix-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M

Run and chat with the model

lemonade run user.tmax-27b-imatrix-MTP-GGUF-IQ2_M

List all available models

lemonade list

🧊 allenai/tmax-27b — imatrix GGUF (2 → 5-bit) + MTP

imatrix + MTP

📦 8.47 – 17.91 GiB IQ2_XS → Q5_K_M · 7 quants ⚡ MTP grafted (Q8) · 95.6% accept · @n=1 🏗️ Qwen3.6-27B derivative 🏅 KLD 0.0014 · top_p 95.1% (Q5_K_M)

🧊 What this is

A bunch of importance-matrix quantizations of allenai/tmax-27b, from aggressive 2-bit up to near-lossless 5-bit, each calibrated with a hybrid importance matrix from real agentic-coding usage logs + wiki text — and each shipping a Multi-Token-Prediction (MTP) draft head bundled in at Q8_0 for built-in speculative decoding (--spec-type draft-mtp). The 2-bit row is benchmarked head-to-head against Qwopus3.6 below; pick a higher-bit row when you have the VRAM.

📉 ~2–6× smaller on disk8.47–17.91 GiB on disk (incl. the bundled MTP head) vs ~54 GiB for FP16. Tuned for English + Python agentic-coding workloads (see calibration scope below).

⚡ MTP speculative decodingGrafted Qwopus3.6 nextn head at Q8_0, 95.6% draft acceptance at n=1 on IQ4_XS. Built-in — no separate draft model required.

🤖 SWE-rebench 70% passIQ2_XS / Q2_K_S / IQ3_M / IQ4_XS all resolve 7/10 SWE-rebench instances — strong even at 2-bit. Full results in §1.

🧰 1. Files & comparison

	Q2_K (plain)	IQ2_XS	IQ2_M	Q2_K_S	IQ3_M	IQ4_XS	Q5_K_M
Technique	plain	hybrid imatrix	hybrid imatrix	hybrid imatrix	hybrid imatrix	hybrid imatrix	hybrid imatrix
File	Q2_K	IQ2_XS	IQ2_M	Q2_K_S	IQ3_M	IQ4_XS	Q5_K_M
Quality	❌	⭐⭐	⭐	⭐⭐	⭐⭐⭐	⭐⭐⭐	⭐⭐⭐
Size (GiB)	9.98	8.47	9.32	9.54	11.72	14.05	17.91
BPW	3.186	2.704	2.976	3.048	3.742	4.486	5.720
PPL (general)	7.6005	25.5923	20.1178	18.1105	20.0037	13.5009	14.1304
KLD med (general)	0.1727	0.1345	0.0767	0.0825	0.0265	0.0059	0.0014
top_p (general)	73.03%	72.89%	78.21%	78.34%	83.72%	91.50%	95.04%

Head-to-head vs Qwopus3.6-27B-Coder

⚔️ IQ2_M head-to-head

Both models are Qwen3.6-27B derivatives benched on the identical general eval corpus and the same llama.cpp build (lower KLD / higher top_p is better). Qwopus3.6 numbers are re-measured here, not quoted from its README, so the comparison is apples-to-apples.

Metric (IQ2_M)	tmax-27b	Qwopus3.6-27B-Coder
KLD med (general)	0.0767	0.0535
top_p (general)	78.21%	83.23%
PPL (general)	20.1178	8.5961

Agentic coding across quants (SWE-rebench)

Every quant run as a coding agent (mini-swe-agent) over the same 10 held-out SWE-rebench instances, one clean Docker container each. pass_rate = fraction whose patch makes the gold FAIL_TO_PASS tests pass; patch_rate = fraction that produced a non-empty diff. Token/step counts are per instance.

Metric	Q2_K	IQ2_XS	IQ2_M	Q2_K_S	IQ3_M	IQ4_XS
pass_rate	50%	70%	60%	70%	70%	70%
patch_rate	100%	100%	100%	100%	100%	100%
resolved	5/10	7/10	6/10	7/10	7/10	7/10
tokens	621,931	784,972	596,658	529,560	770,113	791,474
steps	38.7	49.8	40.9	37.1	47.5	48.3
tool-err	11%	9%	10%	12%	10%	9%

Same 10-instance holdout as the head-to-head, no speculative decoding. At 2-bit the quants stay surprisingly capable agents; higher-bit rows trade size for headroom. Only one repetition is reported above and uncertainties on the pass rate can vary ~5-10% due to sampling variance.

⚠️ These agentic numbers were measured on the pre-recalibration quants (see the "Recalibrated" note in §1). General-eval fidelity is unchanged by the recalibration, so they remain broadly indicative, but a fresh agentic run on the recalibrated GGUFs has not yet been done.

⚠️ Caveat. The 2-bit rows (Q2_K / IQ2_*) are aggressive — strong for their size but rougher on tmax than on Qwopus3.6 (see the head-to-head). Prefer IQ3_M or IQ4_XS when you have the VRAM; reach for the 2-bit rows only when memory is the binding constraint.

📋 Calibration scope — English & Python, agentic coding. The importance matrix (and the windowed packing that shaped it) was calibrated on real agentic-coding sessions that are overwhelmingly English-language and Python-centric (Claude Code, opencode, qwen code). Expect weaker fidelity on other natural languages, non-Python ecosystems, and general-chat workloads.

🔬 2. How they were made

🧮 2.1 Hybrid importance matrix

At 2 bits the quantizer must decide where to spend its limited precision. An importance matrix measures, per input channel, how much each channel drives a layer's output on a calibration corpus, and tells llama-quantize to preserve the high-impact channels. This release blends activation energy E[a²] with weight-column energy ‖W[:, c]‖² · E[a²], collected at ctx=4096. Linear-attention / SSM tensors (Qwen3.6 is a hybrid architecture) pass through with raw E[a²]. The output is a standard GGUF with no runtime overhead.

⚡ 2.2 Bundled MTP (grafted)

tmax-27b's released checkpoint dropped the Qwen3.6 MTP draft head (its mtp_num_hidden_layers flag is vestigial). But tmax and Jackrong/Qwopus3.6-27B-Coder are byte-for-byte architecture-identical finetunes of the same Qwen3.6-27B base (hidden 5120, 64 layers, 24/4 GQA, head_dim 256, vocab 248320, identical hybrid attention layout), so Qwopus3.6's trained nextn head is dimensionally compatible. We graft it onto tmax's trunk as blk.64 and keep it near-lossless at Q8_0 in every quant; llama.cpp serves it as built-in speculative decoding (--spec-type draft-mtp).

Despite being a cross-finetune graft, it drafts at high acceptance — measured on the IQ4_XS trunk over agentic-coding prompts (draft acceptance = fraction of drafted tokens the trunk confirms):

`--spec-draft-n-max`	Draft acceptance
1 (recommended)	95.6%
2	84.7%
3	74.6%
4	74.0%

--spec-draft-n-max 1 is recommended (one nextn layer; higher n drafts longer chains at falling acceptance). Speedup is GPU-bandwidth-dependent — spec decoding helps most on memory-bound CUDA GPUs; the win is smaller on unified-memory/Metal. Acceptance is also prompt-dependent (higher on predictable code, lower on open-ended text).

📚 2.3 Calibration & evaluation data

Calibration and every eval corpus are disjoint by construction — the tool-call eval is the held-out 10% of sessions, windowed exactly like calibration but never seen by it — so §1 measures generalization, not fit. All shipped under calibration_data/.

Corpus	Source	Used for
Calibration	~500k tokens of usage-log text (windowed) + all of `wiki.test.raw`	hybrid imatrix collection
Eval — tools (in-distribution)	held-out logtrain session slice (10%), windowed like calibration but disjoint from it	§1 tools columns (PPL · KLD · top_p)
Eval — general	`combined_en_tiny` (broad English) from `eaddario/imatrix-calibration`	§1 general columns (PPL · KLD · top_p)

🚀 3. Usage

Quick start with Ollama

ollama run hf.co/pearsonkyle/tmax-27b-imatrix-MTP-GGUF:IQ2_M
# also: :Q2_K_S  ·  :IQ2_XS  ·  :Q2_K  ·  :IQ3_M  ·  :IQ4_XS  ·  :Q5_K_M

Building llama.cpp from source (GPU)

apt-get update && apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON   # -DGGML_CUDA=OFF for CPU/Metal
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server
cp llama.cpp/build/bin/llama-* llama.cpp/

MTP needs a recent llama.cpp — --spec-type draft-mtp was merged in 2026-06. Build from current master.

Running the server with MTP speculative decoding

./llama-server \
    --model tmax-27b-IQ4_XS.gguf \
    --ctx-size 16384 --n-gpu-layers 999 \
    --spec-type draft-mtp --spec-draft-n-max 1 \
    --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0 \
    --host 0.0.0.0 --port 1234

Drop the two --spec-* flags to run without MTP. --spec-draft-n-max 1 is optimal (one nextn layer). If the model emits `` reasoning, also pass --chat-template-kwargs '{"enable_thinking": false}' so the answer lands in content.

Querying via the OpenAI-compatible API

import json, urllib.request

def ask(content, max_tokens=256):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # tmax may emit  reasoning. Set enable_thinking False
        # (or raise max_tokens) so the answer lands in "content".
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("Write a Python function that reverses a linked list."))

🪪 4. License & attribution

License: Apache-2.0 — inherited from the base model allenai/tmax-27b. Ai2 asks that use follow its Responsible Use Guidelines.
Base weights: allenai/tmax-27b (Qwen3.6-27B derivative; released checkpoint is the text trunk only — vision + MTP head dropped).
MTP draft head grafted from: pearsonkyle/Qwopus3.6-27B-Coder-imatrix-2bit-MTP-GGUF (also a Qwen3.6-27B finetune; same head it ships natively). Also the head-to-head baseline.
Calibration + quantization performed locally with Quant-Tuner.

Downloads last month: 4,142

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

Model tree for pearsonkyle/tmax-27b-imatrix-MTP-GGUF

Base model

Qwen/Qwen3.6-27B

Finetuned

allenai/tmax-27b

Quantized

(5)

this model