pearsonkyle/Ornith-1.0-9B-imatrix-GGUF

🧊 deepreinforce-ai/Ornith-1.0-9B

imatrix + AWQ + vision

📦 IQ2_M / IQ4_XS / Q5_K_M · 3.4–6.0 GiB 🥇 Q5_K_M near-lossless · KLD 0.002 · top_p 96% 🧪 2-bit AWQ tool-call rescue · param-acc 5%→33% 👁️ vision mmproj (Q8) included 🧠 reasoning · agentic coding

🧊 What this is

a 2 → 5-bit GGUF ladder for deepreinforce-ai/Ornith-1.0-9B — a Qwen3.5-VL–based agentic-coding reasoning model (tool-calling, <think> chain-of-thought). Each quant is calibrated from real agentic-coding usage logs + wiki text, and the calibration method is chosen per bit-width by agentic tool-call fidelity, not just KLD/PPL: the 2-bit ships an AWQ build (activation-aware scaling rescues tool-argument precision that plain 2-bit imatrix loses — param-acc 5%→33%), while 4/5-bit ship plain hybrid imatrix. Q5_K_M is near-lossless (KLD 0.002); IQ4_XS matches it agentically at 20% smaller. The vision tower ships separately as a Q8 mmproj. Plain GGUF, no custom runtime.

📉 2.8× smaller on disk6.0 GiB on disk vs 16.7 GiB for the FP16 text trunk. Tuned for English + Python agentic-coding workloads (see calibration scope below).

👁️ vision retainedA 595 MiB Q8 mmproj (SigLIP-style vision encoder + projector) lets the quantized trunk still accept images via --mmproj.

🧰 1. Files & comparison

Three quants spanning 2 → 5 bit, one variant per bit-width (so ollama run …:IQ2_M / :IQ4_XS / :Q5_K_M all resolve). The 2-bit ships an AWQ build, the 4/5-bit ship plain hybrid imatrix — see the agentic table below for why. FP16 text-trunk reference: 16.69 GiB (not included; fetch from deepreinforce-ai/Ornith-1.0-9B). Vision ships separately as a Q8 mmproj (595 MiB).

File	Method	Size (GiB)	BPW	PPL (gen)	KLD med (gen)	top_p (gen)	KLD med (tools)	top_p (tools)
FP16 (reference)	—	16.69	16.000	5.8874	0.0000	100.0%	0.0000	100.0%
IQ2_M	AWQ + imatrix	3.36	3.223	6.7108	0.1163	79.73%	0.3315	61.09%
IQ4_XS	hybrid imatrix	4.84	4.643	5.9446	0.0096	93.08%	0.0351	74.23%
Q5_K_M	hybrid imatrix	6.02	5.779	5.9759	0.0021	96.41%	0.0083	82.45%

IQ4_XS matches the near-lossless Q5_K_M on agentic tool-calling (below) at 20% smaller — a good default. IQ2_M is the smallest quant that still tool-calls usefully.

🧪 Agentic tool-call fidelity — and why the 2-bit is AWQ

Static KLD/PPL measures how closely the quant's logits track FP16; it does not measure whether the model still uses tools correctly. We replay 25 held-out real agentic-coding sessions (disjoint from calibration) through llama-server, scoring per assistant turn whether the model selects the right tool (tool-sel) and fills the right arguments (param-acc). 3 reps, mean ± σ.

Quant (3.22 / 4.64 / 5.78 bpw)	tool-sel	param-acc	schema-valid
IQ2_M — imatrix (rejected)	0.31 ± .08	0.05 ± .02	0.85
IQ2_M — AWQ (shipped)	0.54 ± .03	0.33 ± .03	0.93
IQ4_XS — imatrix (shipped)	0.65 ± .02	0.36 ± .03	0.98
Q5_K_M — imatrix (shipped)	0.64 ± .02	0.35 ± .02	0.96

At 2 bits the plain-imatrix quant is agentically broken: it emits schema-valid tool calls but fills the wrong arguments (5% param-acc) — even though its static tools-KLD (0.30) is better than the AWQ build's (0.33). AWQ's activation-aware channel scaling rescues argument precision at 2-bit (5% → 33% param-acc, +23 pts tool-sel), so the shipped IQ2_M is the AWQ build. At 4-bit the effect reverses (imatrix leads tool-sel 0.65 vs 0.58 and AWQ's RMSNorm fold slightly perturbs general text), so IQ4_XS/Q5_K_M ship plain imatrix. Takeaway: choose the calibration method by the metric you care about — static fidelity ≠ agentic capability.

Sampling: T=0.25, top_p=0.95, top_k=20, reasoning disabled for eval throughput — absolute scores understate this <think> model, but the relative ranking (the point here) is stable across reps. Harness: Quant-Tuner run_toolcall_reps.py.

📋 Calibration scope — English & Python, agentic coding. The importance matrix (and the windowed packing that shaped it) was calibrated on real agentic-coding sessions that are overwhelmingly English-language and Python-centric, captured from Claude Code, opencode, and qwen code (tool-turn share ≈ 90%). At low bit-widths the codebook's precision is spent where those logs put it: English prompts and Python-flavored tool use (read / edit / bash / grep / write, etc.). Expect weaker fidelity on other natural languages, non-Python ecosystems, and general-chat / vision-heavy workloads.

🤖 SWE-rebench (agentic coding, Q5_K_M)

The Q5_K_M trunk is evaluated on 10 real-world coding issues from nebius/SWE-rebench using the OpenAI Agents SDK pointed at a local llama-server. For each issue the agent gets the problem statement and a live bash tool that shells into a dedicated Docker container with the repo checked out at the failing commit; it reads files, runs tests, and edits code until it produces a git diff or hits the step cap. The patch is graded by actually running the repo's FAIL_TO_PASS suite inside the container — pass/fail is real execution, not fuzzy matching. (Full-loop SWE-rebench on the smaller quants is expensive; the §1 tool-call replay is the lighter agentic proxy used to rank them.)

Metric	Q5_K_M
Issues	10
Patch Rate	100% (10/10)
Pass Rate	30% (3/10)
Max Turns / Timeout	0%
Mean Steps	20.0
Mean Tokens	192K
Tool Error Rate	13.9%
Mean Wall	399s

Sampling: temperature=0.25, top_p=0.95, top_k=20, max_tokens=32768, ctx=131072, thinking=true. Tested on Apple Silicon (Metal) with the SWE-rebench linux/amd64 images under Docker emulation.

---

🔬 2. How they were made

🧮 2.1 Hybrid importance matrix

At low bit-widths the quantizer must decide where to spend its limited precision. An importance matrix measures, per input channel, how much that channel drives each layer's output on a calibration corpus, and tells llama-quantize to preserve the high-impact channels. This release uses a hybrid imatrix blending activation energy E[a²] with weight-column energy ‖W[:, c]²‖ · E[a²], collected at ctx=4096. Linear-attention / SSM tensors (Ornith is a Qwen3.5 hybrid architecture) pass through with raw E[a²]. This is what IQ4_XS and Q5_K_M use. The output is a standard GGUF with no runtime overhead.

🧪 2.1b AWQ for the 2-bit (activation-aware scaling)

At 2 bits, plain imatrix keeps the model's logits close to FP16 but loses the precision to fill correct tool-call arguments (see §1's agentic table: 5% param-acc). AWQ (activation-aware weight quantization) first rescales each input channel by a per-channel factor sc = mean(|xc|)α and folds the inverse into the preceding RMSNorm — mathematically identity in FP16, but it redistributes which channels land on the finer quant levels. The α grid-search and the scale statistics are collected on the same agentic-coding corpus, strided across the whole file so tool-call turns are actually seen. On Ornith's hybrid trunk AWQ scales the full-attention layers' q/k/v + every layer's MLP; the linear-attention layers pass through. Net at 2-bit: param-acc 5% → 33%, tool-sel +23 pts — so IQ2_M ships the AWQ build. (At 4-bit+ the fold's perturbation outweighs the benefit, so those stay imatrix.)

👁️ 2.2 Vision projector (mmproj)

Ornith is a vision-language model. The text trunk is quantized here; its SigLIP-style vision encoder + projector is exported separately as mmproj-Ornith-1.0-9B-Q8_0.gguf (595 MiB, Q8, qwen3vl_merger projector, 27 vision blocks). Pass it with --mmproj alongside any trunk quant to feed images. The projector is small relative to the trunk, so it is kept near-lossless at Q8 while the trunk absorbs the aggressive quantization.

📚 2.3 Calibration & evaluation data

Calibration and every eval corpus are disjoint by construction — both the static tools KLD corpus and the agentic tool-call replay (§1) draw from the held-out 10% of sessions, never seen by calibration — so §1 measures generalization, not fit. All shipped under calibration_data/.

Corpus	Source	Used for
Calibration	~500k tokens of agentic-coding usage-log text (windowed, ≈90% tool-turn) + all of wiki.test.raw	hybrid imatrix collection
Eval — tools (in-distribution)	held-out logtrain session slice (10%), windowed like calibration but disjoint from it	§1 tools columns (KLD · top_p)
Eval — general	combined_en_tiny (broad English) from the eaddario/imatrix-calibration dataset	§1 general columns (PPL · KLD · top_p)

---

🚀 3. Usage

Quick start with Ollama

Each quant is exposed as a tag (the filename's quant suffix):

ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:IQ2_M    # 3.4 GiB · AWQ · smallest that tool-calls
ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:IQ4_XS   # 4.8 GiB · imatrix · best size/quality
ollama run hf.co/pearsonkyle/Ornith-1.0-9B-imatrix-GGUF:Q5_K_M   # 6.0 GiB · imatrix · near-lossless

Building llama.cpp from source (GPU)

apt-get update && apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON   # -DGGML_CUDA=OFF for CPU/Metal
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-server llama-mtmd-cli
cp llama.cpp/build/bin/llama-* llama.cpp/

Running the server (text)

 ./llama-server \
    --model Ornith-1.0-9B-Q5_K_M.gguf \
    --ctx-size 16384 \
    --n-gpu-layers 999 \
    --flash-attn on \
    --host 0.0.0.0 --port 1234

Running with vision (image input)

Pass the bundled Q8 projector with --mmproj:

 ./llama-server \
    --model Ornith-1.0-9B-Q5_K_M.gguf \
    --mmproj mmproj-Ornith-1.0-9B-Q8_0.gguf \
    --ctx-size 16384 --n-gpu-layers 999 --flash-attn on \
    --host 0.0.0.0 --port 1234
# or one-shot on an image:
 ./llama-mtmd-cli -m Ornith-1.0-9B-Q5_K_M.gguf --mmproj mmproj-Ornith-1.0-9B-Q8_0.gguf \
    --image photo.jpg -p "Describe this image."

Querying via the OpenAI-compatible API

import json, urllib.request

def ask(content, max_tokens=1024):
    body = {
        "messages": [{"role": "user", "content": content}],
        "max_tokens": max_tokens,
        # Ornith is a reasoning model: it emits a <think> chain-of-thought before
        # the answer. Give it room, or set enable_thinking False for terse replies.
        "chat_template_kwargs": {"enable_thinking": True},
    }
    req = urllib.request.Request("http://127.0.0.1:1234/v1/chat/completions",
                                 json.dumps(body).encode(),
                                 {"Content-Type": "application/json"})
    return json.loads(urllib.request.urlopen(req).read())["choices"][0]["message"]["content"]

print(ask("Write a Python function that reverses a linked list."))

---

🪪 4. License & attribution

• Licensed MIT, inherited from the base model deepreinforce-ai/Ornith-1.0-9B. Confirm the current terms on the base model card before redistributing.
• Base weights: deepreinforce-ai/Ornith-1.0-9B — a Qwen3.5-VL–based agentic-coding reasoning model (post-trained on Qwen 3.5 + Gemma 4).
• Calibration + quantization performed locally with Quant-Tuner; vendored llama.cpp at commit f3e1828. The Q5_K_M quant was produced with a hybrid imatrix on the calibration corpora below.
• Calibration data (usage logs) scraped using LogMiner.