hipfire-models/ornith1.5-35b-a3b

Ornith-1.5-35B-A3B — hipfire

hipfire quantization of ornith-ai/Ornith-1.5-35B-A3B: a 35B-A3B hybrid MoE (30 linear-attention + 10 full-attention layers, 256 experts, top-8) with a vision tower and a multi-token-prediction draft head.

File	Size	What
ornith1.5-35b-a3b.mq4	17.7 GiB	Text trunk, MQ4G256V2 (qt44)
ornith1.5-35b-a3b.mtp	462 MiB	MTP draft head — replacement, see below
ornith1.5-35b-a3b.vl	865 MiB	Vision tower, F16

Needs ~22 GB of VRAM with the default Q8 KV cache.

The MTP head is NOT the one from the base model

This repository ships a third-party trained MTP head, not Ornith 1.5's. That is a deliberate substitution and you should know about it.

The head in the official Ornith 1.5 release drafts poorly. Its mtp.* tensors carry a standard deviation of exactly 0.0200 — the config's initializer_range — with clean Gaussian statistics, and it is discussed upstream in ornith-ai/Ornith-1.5-35B-A3B discussion #10, which reports ~13% acceptance. We measured the same behaviour independently in two engines: hipfire and llama.cpp both accept the first drafted token at 0.82–0.93 and then collapse (hipfire 0.29 at depth 2, llama.cpp 0.15). With that head, speculative decoding is slower than plain autoregressive decode.

So the shipped .mtp is built from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY, a head warm-started from Qwen3.6's MTP weights and KL-distilled against Ornith 1.5 hidden states.

Per-position draft acceptance, gfx1151, greedy, code prompt:

Head	depth-1	depth-2	depth-3
Ornith 1.5 native head	0.823	0.292	0.179
This head	0.877	0.620	0.528

Speculative decoding is lossless with respect to the trunk regardless of head quality — the trunk verifies every drafted token — so a weak head costs throughput, never correctness.

If you want the vendor's original head instead, extract it from the base checkpoint yourself; it is not redistributed here.

Throughput

Measured on a Strix Halo (gfx1151, Radeon 8060S), 256 generated tokens, greedy, reasoning off, Q8 KV, code-editing prompt. Medians of repeated runs on an otherwise idle machine.

Configuration	tok/s	τ	accept
Autoregressive (no speculation)	~70	—	—
MTP, K=3	77.4	2.5	0.61
MTP + ngram-modulated drafting	~150	5.43	0.937

The ngram-modulated arm is the large win — roughly 2.1× over autoregressive decode. It engages only when reasoning is disabled (max_think_tokens = 1).

Numbers are hardware- and prompt-specific. Prose prompts accept far fewer drafted tokens than code; treat these as an upper band, not a guarantee.

Quantization

The trunk is MQ4G256V2 (qt44): 4-bit weights in 256-element groups with an FWHT rotation and two fp16 scale/zero pairs per group. Attention tensors in the first and last two layers are promoted to MQ6G256; lm_head, embeddings, MoE routers and shared-expert gates stay Q8. The vision tower is F16.

The draft head is qt44 with a Q8 lm_head_draft compressed to a 16,384-token draft vocabulary (the trunk's own lm_head still verifies against the full 248,320-token vocabulary, so the compression cannot change what is committed).

Known headroom: the fixed tier (lm_head in particular, at Q8) is a large share of per-token decode bytes. Rebuilding with the mq4r recipe (--no-q8-router) is expected to recover a substantial part of the autoregressive gap against comparable SKUs; it has not been done here because it changes the shipped artifact and needs its own quality validation.

Usage

hipfire run ornith1.5:35b-a3b

HIPFIRE_MTP_K=3 is the measured sweet spot on gfx1151 (K=4 75.5, K=5 72.3, K=6 73.5 tok/s). Recommended sampling follows the base model: temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5.

Licensing

This repository is mixed-licence. See NOTICE for the full chain.

• Trunk and vision tower derive from ornith-ai/Ornith-1.5-35B-A3B, which declares MIT in its repository metadata. Upstream ships no LICENSE file and its own license_link returned 404 when this artifact was built; the canonical MIT terms are reproduced in LICENSE-MIT for reference, and copyright remains with the upstream authors.
• MTP draft head derives from shisa-ai/Ornith-1.5-35B-A3B-MTP-ONLY (Apache-2.0, LICENSE-APACHE-2.0), itself a modified derivative of the MTP head from Qwen/Qwen3.6-35B-A3B (Apache-2.0).

hipfire's contribution is quantization only: conversion to qt44 with FWHT rotation, the selective MQ6/Q8 tiers, and the compressed draft vocabulary. No warranty; quantization is lossy.