Vontra/Qwen3.8-Flash-Next-MLX-8bit-MTP

Qwen3.8 Flash Next — MLX 8-bit with native MTP

A uniform 8-bit MLX conversion of Qwen/Qwen3.8-Flash-Next, rebuilt directly from the official BF16 checkpoint with the model's native MTP draft block preserved.

Original model · Qwen overview · MLX-VLM · Qwen Community License 1.0

About this conversion

This repository contains the standard uniform 8-bit release. Eligible language-model and native-MTP modules use MLX affine quantisation at group size 32. Multimodal modules and MoE router gates remain in BF16. Group size 32 also covers the model's 160-wide hashed n-gram embedding tables.

Item	Value
Repository	Vontra/Qwen3.8-Flash-Next-MLX-8bit-MTP
Base model	Qwen/Qwen3.8-Flash-Next
Source weights	Official BF16 checkpoint
Format	MLX safetensors
Quantisation	Uniform 8-bit affine
Group size	32
Retained in BF16	Multimodal modules and mlp.gate router modules
Native MTP	Included, one Qwen4Exp draft block
Indexed tensors	3,747 total, including 76 MTP entries
Weight shards	42
Weight size	202.951 GB / 189.013 GiB
Configured context	262,144 tokens
Architecture	qwen4_exp vision-language sparse MoE

The upstream tokenizer, chat template, vision processor, generation configuration, licence, and native MTP configuration are retained.

Use a runtime with explicit qwen4_exp and native-MTP support. A runtime that does not construct the Qwen4Exp MTP module may reject the 76 MTP tensor entries during strict loading.

Download and use

python -m pip install --upgrade huggingface_hub

hf download Vontra/Qwen3.8-Flash-Next-MLX-8bit-MTP \
  --local-dir ./Qwen3.8-Flash-Next-MLX-8bit-MTP

In a compatible oMLX build:

1. Add the downloaded directory to the oMLX model directories.
2. Refresh the model registry and select Qwen3.8-Flash-Next-MLX-8bit-MTP.
3. Enable native MTP and set the draft depth to three tokens.
4. Load the model and start a chat or use the OpenAI-compatible API.

The model can run with native MTP disabled for baseline or troubleshooting comparisons. MTP changes the decode path, not the model's intended greedy output.

Apple M3 Studio performance

Benchmark environment: oMLX 0.6.3rc3 (build 2475) on an Apple M3 Studio. Measurements use greedy decoding, seed 6330, identical prompts, a separate warm-up, and 3 measured 512-token runs per mode.

Runtime mode	Runs	Output per run	Median generation speed	Drafted	Accepted	Acceptance
Native MTP disabled	3	512 tokens	20.0693 tokens/s	Not applicable	Not applicable	Not applicable
Native MTP enabled, three draft tokens	3	512 tokens	24.8956 tokens/s	1004	711	70.82%

The measured throughput delta with MTP enabled was 24.05%. Exact-output parity: Passed. All sustained runs reached 512 generated tokens: Passed. Coherence checks: Passed. MTP telemetry capture: Passed.

Draft acceptance is reported beside throughput because a high acceptance rate does not by itself prove a speed improvement. Results vary with prompt length, context growth, cache state, runtime version, memory pressure, and thermal conditions. The first request after loading includes model and kernel warm-up and is excluded from the steady-state result.

Runtime compatibility

Component	Release used	Validation scope	Status
oMLX	0.6.3rc3 (build 2475)	Model load, text generation, native MTP, parity, and sustained decode	Passed
MLX-VLM	0.6.3	Qwen4Exp construction, conversion, and strict weight loading	Passed
MLX	0.32.0	Quantised tensor backend and generation kernels	Passed

Older builds without explicit qwen4_exp native-MTP support are not compatible with this checkpoint. Vision input support also depends on the selected MLX-VLM and oMLX versions; the performance figures above cover text generation only.

Architecture

Qwen3.8 Flash Next is a causal language model with a vision encoder. Its language stack combines Gated DeltaNet, Qwen Sparse Attention, sparse mixture-of-experts layers, widened gated residual streams, hashed bigram and trigram embeddings, and a native next-token-prediction block for speculative decoding.

Architecture detail	Upstream value
Language-model parameters	125B total / 6B active
N-gram embedding	51B parameters, 20,000,000 entries
Native MTP	4B parameters, one draft layer
Hidden size	2,560
Token embedding / LM output	248,320 padded
Layers	48
Layer layout	12 × (3 × Gated DeltaNet + 1 × Qwen Sparse Attention), each followed by MoE
Gated DeltaNet heads	48 V / 16 QK, head dimension 128
Sparse-attention heads	24 Q / 2 KV, head dimension 256
Sparse-attention budget	512 blocks / 2,048 tokens
Routed / active experts	512 / 10, plus 1 shared expert
Expert intermediate size	640
Gated-residual branches / bottleneck	4 / 320
Native context	262,144 tokens, extensible upstream to 1,000,000

For upstream evaluations, intended use, limitations, safety guidance, and the full architecture discussion, see the original model card.

Conversion and validation

• The converter read the official BF16 checkpoint directly.
• Eligible modules were quantised uniformly at 8 bits using affine quantisation and group size 32.
• Multimodal modules and MoE router gates were retained in BF16.
• Structural validation of all 3,747 indexed tensors, all 42 weight shards, and all 76 native-MTP tensor entries: Passed.
• Exact deterministic MTP-off/on parity: Passed.
• 3 sustained 512-token runs per mode: Passed.
• MTP drafted/accepted telemetry: Passed.
• Coherent long-generation checks: Passed.

This is a community conversion, not an official Qwen release.

Limitations

• Quantisation can reduce quality relative to BF16. The effect is usually strongest at 2-bit precision.
• The native 262,144-token context does not guarantee that every Apple-silicon system can prefill or decode that length within available unified memory.
• Native MTP needs a compatible runtime and may not improve every prompt or context length.
• The checkpoint is an MLX release for Apple silicon. It is not a GGUF, CUDA, TensorRT-LLM, or vLLM checkpoint.
• Upstream model limitations and safety considerations still apply.

Licence and attribution

The upstream model is released under the Qwen Community License 1.0. The required licence text is included in this repository and should be reviewed before use or redistribution.

Model design, training, evaluations, and upstream documentation belong to Qwen and the original contributors. The MLX conversion, native-MTP preservation, Apple-silicon validation, and packaging are provided by Vontra.