programmer-666/Ornith-1.0-35B-MLX-oQ7-mtp

Ornith-1.0-35B-oQ7-mtp

MLX format quantization of deepreinforce-ai/Ornith-1.0-35B, produced with mlx-optiq and shipped with a grafted Multi-Token Prediction (MTP) head for speculative decoding on Apple Silicon.

Model Details

Model Description

This repository contains a mixed precision MLX quantization of Ornith-1.0-35B, a Qwen3.5-35B-A3B based mixture of experts model. The quantization was produced with mlx-optiq, which performs a per-layer KL sensitivity analysis against the original BF16 checkpoint and assigns each layer either 6-bit or 8-bit precision so that the weighted average lands at a target of 7.5 bits per weight (BPW). Layers that are more sensitive to quantization error are kept at 8-bit, while more robust layers are reduced to 6-bit.

An auxiliary MTP head, grafted from mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit, is included as mtp.safetensors. It is used as a self-speculative draft model during decoding and is architecturally compatible because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share the same hidden size, layer count, expert count, and block layout, and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.

• Developed by: programmer-666
• Model type: Causal decoder only mixture of experts language model (Qwen3.5-35B-A3B architecture, qwen35moe, 40 layers)
• Language(s): Inherited from the base model
• License: Apache 2.0 for this repository. See the Licenses section below for the licenses of the underlying components.
• Quantized from model: deepreinforce-ai/Ornith-1.0-35B
• MTP head source: mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit

Model Sources

• Base model: https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B
• Quantization tool: https://mlx-optiq.com
• Serving engine used for benchmarks: https://github.com/jundot/omlx

Model Properties

Property	Value
Base model	deepreinforce-ai/Ornith-1.0-35B
Architecture	Qwen3.5-35B-A3B (qwen35moe, 40 layers)
Total parameters	35B
Active parameters per token	3B
Quantization	mlx-optiq mixed precision (6 bit / 8 bit)
Target BPW	7.5
Achieved BPW	7.75
Layers at 6-bit	29
Layers at 8-bit	362
Group size	32
Calibration sequences	40
MTP head source	mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit
Format	MLX safetensors

Uses

Direct Use

This model is intended for local text generation on Apple Silicon devices using MLX based inference engines such as mlx-lm or optiq serve. It is suited for users who want most of the quality of the BF16 checkpoint at roughly half the memory footprint, with an optional speculative decoding path for faster generation.

Out-of-Scope Use

This is a quantized derivative of a third party base model. It has not been independently evaluated for safety, factuality, or fitness for any particular downstream task. It should not be used in high stakes settings (medical, legal, financial, or safety critical decisions) without additional evaluation. Refer to the base model card for its intended use cases and known limitations, since these are inherited by this quantization.

Bias, Risks, and Limitations

Quantization can shift a model's behavior relative to the original weights, even when overall benchmark scores are similar. Mixed precision quantization at 7.5 BPW is expected to be close to BF16 quality, but no independent evaluation of downstream task accuracy, factuality, or bias has been performed for this specific quantized artifact. Users should treat outputs as they would from the base model and are encouraged to run their own evaluations for their use case before deploying it in production.

The intelligence benchmark results reported below use 30-question samples from each dataset and should be treated as indicative rather than definitive. Full dataset evaluations may yield different figures.

How to Get Started with the Model

With mlx-optiq serve (enables MTP speculative decoding)

optiq serve \
  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
  --mtp \
  --port 8080

With mlx-lm

mlx_lm.generate \
  --model programmer-666/Ornith-1.0-35B-oQ7-mtp \
  --prompt "Your prompt here"

Note: MTP is not available through mlx_lm.generate. Use optiq serve or omlx if you want speculative decoding.

Training Details

This repository does not modify the base model's weights beyond quantization; no additional fine-tuning was performed. For training data and training procedure, refer to the base model card.

Quantization Procedure

Quantization was performed with mlx-optiq, which runs a per-layer KL sensitivity analysis to assign bit widths. Layers with higher sensitivity to quantization error retain 8-bit precision, while more robust layers are assigned 6-bit precision. The reference model used during sensitivity calibration was the original BF16 checkpoint.

Conversion command:

optiq convert deepreinforce-ai/Ornith-1.0-35B \
  --candidate-bits 6,8 \
  --target-bpw 7.5 \
  --reference bf16 \
  --n-calibration 40 \
  --group-size 32 \
  --skip-baselines \
  -o Ornith-1.0-35B-oQ7-mtp

Multi-Token Prediction (MTP)

The mtp.safetensors file contains an auxiliary prediction head grafted from mlx-community/Qwen3.6-35B-A3B-OptiQ-4bit. This works because Qwen3.5-35B-A3B and Qwen3.6-35B-A3B share an identical structure (hidden dimension 2048, 40 layers, 256 experts, same block layout), and Ornith is fine-tuned directly from Qwen3.5-35B-A3B.

MTP uses this auxiliary head as a draft model for speculative decoding, giving roughly a 1.3x to 1.4x decode speedup on Apple Silicon during greedy generation.

Hardware Requirements

Configuration	Notes
Recommended	Apple Silicon with 64GB or more of unified memory
Tested on	M4 Max MacBook Pro, 128GB
Disk space	Approximately 37GB

Evaluation

Testing Setup

All benchmarks were run locally on an M4 Max MacBook Pro (128GB unified memory) using oMLX, an LLM inference engine optimized for Apple Silicon. Three models were evaluated side by side:

• Ornith-1.0-35B-oQ7-mtp (this repository)
• Ornith-1.0-35B-bf16 (original BF16 checkpoint, reference)
• Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for context)

Performance figures come from single request runs unless noted as continuous batching. Intelligence benchmarks were run with thinking mode enabled and use 30-question samples drawn randomly from each dataset; they are indicative and should not be compared directly to full-dataset evaluations published elsewhere.

Intelligence Benchmarks

All results are for this model (Ornith-1.0-35B-oQ7-mtp) with thinking enabled.

Benchmark	Sampled	Correct	Accuracy	Time (s)
MMLU	30 / 14042	26 / 30	86.7%	735.6
MMLU Pro	30 / 12032	23 / 30	76.7%	986.4
HellaSwag	30 / 10042	26 / 30	86.7%	441.4
TruthfulQA	30 / 817	27 / 30	90.0%	605.8
ARC Challenge	30 / 1172	28 / 30	93.3%	356.5
Winogrande	30 / 1267	26 / 30	86.7%	338.3
GSM8K	30 / 1319	29 / 30	96.7%	810.7
MathQA	30 / 2985	28 / 30	93.3%	1073.7
HumanEval	30 / 164	27 / 30	90.0%	1139.9
MBPP	30 / 500	27 / 30	90.0%	1516.2
LiveCodeBench	30 / 1055	19 / 30	63.3%	4501.8
BBQ	30 / 10864	28 / 30	93.3%	233.9
SafetyBench	30 / 11435	26 / 30	86.7%	270.2

These results represent 30-question random samples from each dataset evaluated with thinking mode enabled. Sample-based scores carry higher variance than full dataset evaluations and may not reflect performance on the full benchmark. Results are not directly comparable to published leaderboard figures which typically use the full dataset without thinking mode.

Performance Benchmarks

Ornith-1.0-35B-oQ7-mtp (this repository)

Test	TTFT (ms)	TPOT (ms)	pp TPS	tg TPS	E2E (s)	Throughput	Peak Mem
pp1024/tg128	787.3	10.70	1300.6 tok/s	94.2 tok/s	2.147	536.6 tok/s	37.36 GB
pp4096/tg128	2541.2	10.96	1611.9 tok/s	91.9 tok/s	3.934	1073.8 tok/s	38.14 GB
pp8192/tg128	5421.8	11.35	1510.9 tok/s	88.8 tok/s	6.863	1212.3 tok/s	38.48 GB
pp16384/tg128	12895.5	12.89	1270.5 tok/s	78.2 tok/s	14.533	1136.2 tok/s	39.10 GB
pp32768/tg128	29809.2	13.68	1099.3 tok/s	73.7 tok/s	31.546	1042.8 tok/s	40.44 GB
pp65536/tg128	94152.4	25.95	696.1 tok/s	38.8 tok/s	97.449	673.8 tok/s	43.13 GB
pp131072/tg128	348440.0	30.93	376.2 tok/s	32.6 tok/s	352.368	372.3 tok/s	48.50 GB
pp200000/tg128	726434.2	36.19	275.3 tok/s	27.9 tok/s	731.030	273.8 tok/s	54.17 GB

Continuous batching, pp1024/tg128:

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT (ms)	E2E (s)
1x	94.2 tok/s	1.00x	1300.6 tok/s	1300.6 tok/s	787.3	2.147
2x	110.1 tok/s	1.17x	549.5 tok/s	274.8 tok/s	3726.8	6.052
4x	125.5 tok/s	1.33x	826.0 tok/s	206.5 tok/s	4836.1	9.038

Ornith-1.0-35B-bf16 (reference)

Test	TTFT (ms)	TPOT (ms)	pp TPS	tg TPS	E2E (s)	Throughput	Peak Mem
pp1024/tg128	834.6	16.17	1227.0 tok/s	62.3 tok/s	2.888	398.9 tok/s	65.62 GB
pp4096/tg128	2269.3	16.46	1805.0 tok/s	61.2 tok/s	4.359	969.0 tok/s	66.40 GB
pp8192/tg128	4984.5	15.94	1643.5 tok/s	63.2 tok/s	7.009	1187.1 tok/s	66.73 GB
pp16384/tg128	11321.8	16.73	1447.1 tok/s	60.2 tok/s	13.447	1228.0 tok/s	67.39 GB
pp32768/tg128	28415.1	18.08	1153.2 tok/s	55.7 tok/s	30.711	1071.1 tok/s	68.70 GB
pp65536/tg128	85199.7	30.94	769.2 tok/s	32.6 tok/s	89.130	736.7 tok/s	71.35 GB
pp131072/tg128	346498.2	36.68	378.3 tok/s	27.5 tok/s	351.157	373.6 tok/s	76.79 GB
pp200000/tg128	630686.4	40.49	317.1 tok/s	24.9 tok/s	635.829	314.8 tok/s	82.47 GB

Continuous batching, pp1024/tg128:

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT (ms)	E2E (s)
1x	62.3 tok/s	1.00x	1227.0 tok/s	1227.0 tok/s	834.6	2.888
2x	43.1 tok/s	0.69x	511.9 tok/s	255.9 tok/s	4000.4	9.941
4x	71.0 tok/s	1.14x	938.8 tok/s	234.7 tok/s	4212.9	11.578

Qwen3.6-35B-A3B-OptiQ-4bit (MTP head source, shown for reference)

Test	TTFT (ms)	TPOT (ms)	pp TPS	tg TPS	E2E (s)	Throughput	Peak Mem
pp1024/tg128	765.7	9.03	1337.3 tok/s	111.6 tok/s	1.913	602.3 tok/s	21.72 GB
pp4096/tg128	2466.8	9.38	1660.5 tok/s	107.5 tok/s	3.658	1154.7 tok/s	22.49 GB
pp8192/tg128	5296.1	9.66	1546.8 tok/s	104.4 tok/s	6.523	1275.6 tok/s	22.83 GB
pp16384/tg128	12197.2	10.30	1343.3 tok/s	97.9 tok/s	13.505	1222.6 tok/s	23.46 GB
pp32768/tg128	30598.7	11.70	1070.9 tok/s	86.2 tok/s	32.084	1025.3 tok/s	24.80 GB
pp65536/tg128	87223.7	18.90	751.4 tok/s	53.3 tok/s	89.624	732.7 tok/s	27.48 GB
pp131072/tg128	343907.5	27.19	381.1 tok/s	37.1 tok/s	347.361	377.7 tok/s	32.86 GB
pp200000/tg128	675070.1	32.83	296.3 tok/s	30.7 tok/s	679.240	294.6 tok/s	38.52 GB

Continuous batching, pp1024/tg128:

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT (ms)	E2E (s)
1x	111.6 tok/s	1.00x	1337.3 tok/s	1337.3 tok/s	765.7	1.913
2x	148.6 tok/s	1.33x	569.3 tok/s	284.6 tok/s	3597.4	5.320
4x	161.0 tok/s	1.44x	873.2 tok/s	218.3 tok/s	4564.0	7.870

Summary

Compared to the BF16 checkpoint, this quantization roughly halves peak memory usage (37 to 54 GB versus 66 to 82 GB depending on context length) while producing 51% faster token generation at standard context lengths (94.2 tok/s versus 62.3 tok/s at pp1024/tg128). The BF16 model's continuous batching throughput degrades at batch size 2 (0.69x), while this quantization scales consistently to 1.33x at batch size 4. The 4-bit Qwen3.6-35B-A3B-OptiQ-4bit model is faster and lighter on its own; it is included here as the MTP draft head source and as a context point for the tradeoff between model size and throughput.

Licenses

• Ornith-1.0-35B (base model): MIT License
• Qwen3.5-35B-A3B (base architecture): Apache 2.0
• Qwen3.6-35B-A3B (MTP head source): Apache 2.0
• This repository (quantized weights and MTP head): Apache 2.0

Users should review the license terms of each underlying component before use or redistribution.

Citation

If you use this model, please cite the original Ornith-1.0-35B model and the mlx-optiq quantization tool.

@misc{ornith-1.0-35b-oq7-mtp,
  title = {Ornith-1.0-35B-oQ7-mtp},
  author = {programmer-666},
  year = {2026},
  note = {MLX mixed precision quantization of deepreinforce-ai/Ornith-1.0-35B with grafted MTP head},
  howpublished = {\url{https://huggingface.co/programmer-666/Ornith-1.0-35B-oQ7-mtp}}
}

Model Card Contact

For questions about this quantization, open a discussion on this repository's Community tab. For questions about the base model, refer to deepreinforce-ai/Ornith-1.0-35B.