Qwythos-27B-v1-NVFP4-MTP

NVFP4 quantisation of empero-ai/Qwythos-27B-v1 built with NVIDIA TensorRT Model Optimizer (ModelOpt), with the native MTP speculative-decoding head restored and measured working at num_speculative_tokens: 3.

All credit for the model belongs to Empero — Qwythos-27B-v1 is their open-weight, full-parameter reasoning model, released un-ablated with its native multi-token-prediction head, full vision tower and long-context configuration intact. Read their card; it is worth it.

This repo only makes that model practical on smaller hardware: ~55.6 GB bf16 → 20.6 GB NVFP4, quantised and load-verified on an NVIDIA DGX Spark (GB10, sm_121a).

There is another public NVFP4 of this model, CocaKova/Qwythos-27B-v1-NVFP4, built with llm-compressor (compressed-tensors). It is a careful build and it came first. This one is a ModelOpt build, which is a different quantisation toolchain and a different kernel path in vLLM. Measurements against their published figures are below. Pick on the numbers, not the badge.

What is quantised, and what is not

Component	Precision
Text backbone Linear layers (64 layers, hybrid Gated-DeltaNet + full attention)	NVFP4 (FP4 weights + activations, group size 16)
lm_head	bf16
Vision tower (all 333 model.visual.* tensors)	bf16
Gated-DeltaNet conv1d layers	bf16
Native MTP head (all 15 mtp.* tensors)	bf16
KV cache	FP8 (e4m3) scales exported

ModelOpt excluded the vision tower, lm_head and the conv1d layers automatically — 1699 quantizers were inserted, and none of them landed on those. Calibrated on 512 samples at 2048 tokens (cnn_dailymail).

The MTP head had to be rebuilt, and it is worse than the usual warning

The known trap is that transformers silently drops mtp.* tensors on load (_keys_to_ignore_on_load_unexpected), so a naive quantize-and-save loses the speculative-decoding head. On this model it dropped all fifteen — the export came out with mtp tensor count zero, not merely missing the fusion projection.

All 15 tensors (849.4 MB) were copied verbatim from the source checkpoint, re-attached through the safetensors index, and — this part matters — added to the quantization exclusion list by MODULE name:

mtp.fc            not  mtp.fc.weight
mtp.norm          not  mtp.norm.weight
mtp.layers.0.self_attn.q_proj      ... and so on

The tensor-name form matches no module, so the exclusion silently does nothing, vLLM reads full-size bf16 into an FP4-packed parameter, and the server crash-loops on assert param_data.shape == loaded_weight.shape while the build reports success. If you are reproducing this yourself, that is the failure to expect.

Measured acceptance after quantisation: 0.905 at 2K context, 0.683 at 72K, at num_speculative_tokens: 3 under vLLM — read from vLLM's own spec_decode_num_accepted_tokens_total / ..._draft_tokens_total counters, not inferred.

Measured performance

DGX Spark (GB10, 128 GB unified), vLLM 0.25.2, NVFP4 + MTP k=3, single stream, max_model_len 131072. Token counts from usage.completion_tokens — never from streamed chunk counts, which under speculative decoding under-report by the tokens-per-chunk factor (here 2.7–3.6×).

Served at the full 262,144 context, 8 concurrent (vLLM's own boot line reports Maximum concurrency for 262,144 tokens per request: 8.57x; KV pool 2,246,097 tokens):

Prompt context	Decode	MTP acceptance	Tokens/chunk
2,727 tok	28.1 tok/s	0.949	3.77
20,727 tok	20.2 tok/s	0.615	2.78
93,627 tok	17.0 tok/s	0.544	2.61

Serving the same build at 131,072 instead measured 26.9 / 19.9 / 19.8 tok/s — i.e. the 262k window costs essentially nothing at short and mid context and about 14% at 72k. There is no reason to give up the native window for speed.

For reference, the CocaKova card reports 16.2 / 16.1 / 14.0 tok/s at its 2K / 16K / 72K points for the compressed-tensors build. Those runs are not ours and the prompts differ, so treat this as indicative rather than a controlled comparison — but the builds are in the same range and this one is not slower.

Caveats, because single-run numbers deserve them: one machine, one run per point, no concurrency sweep. The GPU on this box is clock-capped at 2000 MHz — decode is memory-bandwidth bound and measured flat across a 34% clock range, so the decode figures above are unaffected, but TTFT would be roughly 10% better on an unclocked Spark. No quality evaluation of the 4-bit result against the bf16 source was run here.

One measurement note that cost real time here: decode on this hardware is memory-bandwidth bound, so anything else touching memory bandwidth wrecks the numbers. A large concurrent upload dragged the same request from 28 tok/s down to 14. If you benchmark this model, benchmark it on an otherwise idle box.

DFlash was tried, and MTP won — so use MTP

DFlash is the other speculative method available for this family. There is no DFlash draft trained for Qwythos, so the nearest thing was tested: z-lab/Qwen3.6-27B-DFlash, which is dimensionally compatible (hidden 5120, vocab 248320, identical to this model). It loaded first try and worked — a draft trained against a different base still reached 0.752 acceptance on short prompts, which says something about how close Qwythos sits to its Qwen3.6 relatives.

It still loses:

context	MTP k=3	borrowed DFlash k=3
2K	28.1 tok/s · accept 0.949	29.6 tok/s · accept 0.752
16K	20.2 · 0.615	18.8 · 0.361
72K	17.0 · 0.544	16.9 · 0.360
KV pool	2,246,097 tokens	1,761,015 (−18%)

DFlash edges ahead at short context and loses everywhere else, while its separate draft cache costs 18% of the KV pool. Acceptance roughly halves past 16k — the signature of a draft that is a decent general match but was never trained on this model's long-range behaviour. Use the MTP head; it is already in the checkpoint and it is free.

A purpose-built DFlash draft would likely do better — for comparison, a dedicated draft on another model runs 0.448 acceptance and beats MTP on code work. That is a training project, not a config change.

Context length

This repo ships the native 262,144-token configuration: max_position_embeddings 262144 with no rope_scaling.

The upstream checkpoint declares 1,048,576 positions via static YaRN (factor: 4.0 over an original 262144). Two reasons this build does not:

1. Static YaRN applies at every length, including short prompts, and slightly degrades short-context quality — a tax on the common case to serve the rare one.
2. 262,144 is genuinely usable here, not a compromise: this box reports a 2,138,983-token KV cache, i.e. 8.1 concurrent requests at 262k, against 2.0 at 1M.

You do not need to edit anything to get the full 1M window — pass it at serve time:

--hf-overrides '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144},"max_position_embeddings":1048576}'

Usage (vLLM)

vllm serve PassingByPixels/Qwythos-27B-v1-NVFP4-MTP \
    --trust-remote-code \
    --max-model-len 131072 \
    --kv-cache-dtype fp8 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3

Sampling, from the upstream generation_config.json: temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05.

Drop --speculative-config to run without speculation. If you do not want the MTP head at all, PassingByPixels/Qwythos-27B-v1-NVFP4 is the same quantisation without it (849 MB smaller).

Limitations

• 4-bit quantisation is lossy. No quality evaluation against the bf16 source was run here — the performance numbers above are throughput only.
• Vision was frozen upstream and the tower is preserved in bf16, but image behaviour is untested post-quantisation.
• The VL processor configs (preprocessor_config.json, video_preprocessor_config.json) and the chat template are not emitted by ModelOpt and were copied verbatim from Empero's own checkpoint, so they match the source exactly.
• Qwythos is intentionally uncensored for technical and research use — see the original card. Apply your own application-level controls.
• This is the pre-RL v1 checkpoint.

Credits

• Empero — the model, the training curriculum, and the decision to ship it un-ablated.
• Qwen team — the base.
• CocaKova — the first public NVFP4 of this model, and a card whose MTP-surgery warning saved time here.
• NVIDIA — TensorRT Model Optimizer.

Quantised and load-verified by PassingByPixels on a DGX Spark. Apache-2.0, same as the original.