Frontis-MA1-35B — NVFP4 (with MTP)

This is an NVFP4 quant of FrontisAI/Frontis-MA1-35B, built and tested on an NVIDIA GB10 (ASUS Ascent GX10, 121 GiB unified memory, CUDA 13.0) running vLLM.

I run a lot of models on Blackwell hardware, and NVFP4 is the format that actually pays for itself there — the tensor cores do FP4 natively, so you get the size reduction without a dequant tax. Frontis-MA1 is an execution-grounded MLE/AutoResearch model I wanted on my own boxes and there was no NVFP4 build of it anywhere, so I made one. As far as I can tell this is the first.

Short version: 67 GB of BF16 becomes 23.3 GiB, it serves on vLLM out of the box, and with the MTP layers grafted back in it runs at 57.4 tok/s on a single GB10 — a measured 1.35× over the same checkpoint without speculative decoding.

The MTP layers are in here, and that took an extra step

The base ships 19 mtp.* tensors — a full multi-token-prediction block inherited from Qwen3.6-35B-A3B. llm-compressor drops them. It loads the model through a class that never instantiates the MTP layers, so they are not quantized badly, they simply never reach the output. A first pass gave me a checkpoint with zero MTP tensors and no speculative decoding at all.

So I grafted them back: the 19 tensors are copied in from the base at BF16 and added to quantization_config.ignore. They are deliberately not quantized — a drafter with degraded numerics produces bad drafts and tanks the acceptance rate, which costs you more than the 1.6 GiB it saves.

If you re-quantize this model yourself, check your output for mtp.* tensors before you conclude MTP "doesn't work" on it.

What you need

NVFP4 is a compressed-tensors format and needs a recent vLLM on Blackwell-class hardware (SM120/SM121 — RTX 50-series, B200, GB10/GB200). It will not run on Ampere or Ada, and it is not a GGUF, so llama.cpp / Ollama / LM Studio cannot load it.

Tested on vllm/vllm-openai:nightly-aarch64, vLLM 0.26.1rc1.

Files

File	Size
`model.safetensors`	23.32 GiB
`config.json`, tokenizer, `chat_template.jinja`, `processor_config.json`	small
`recipe.yaml`, `NVFP4_QUANT_RECEIPT.txt`	exact quant parameters, so you can reproduce it

Single shard. 124,325 tensors — 4-bit weights packed as U8 with F8_E4M3 group scales, and everything I deliberately left alone (vision tower, router, embeddings, MTP block) still BF16.

Running it

docker run -d --name vllm-frontis --gpus all --network host \
  --ipc=host --shm-size=16g -v /path/to/models:/models \
  -e CUTE_DSL_ARCH=sm_121a \
  vllm/vllm-openai:nightly-aarch64 \
    /models/Frontis-MA1-35B-NVFP4 \
    --served-model-name frontis-ma1-35b --port 8000 --trust-remote-code \
    --max-model-len 65536 --max-num-seqs 2 \
    --gpu-memory-utilization 0.38 --kv-cache-dtype fp8 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Three things cost me time before I worked them out:

The image entrypoint is already ["vllm","serve"]. The first argument is the model path. Pass the word serve and you get vllm serve serve /models/..., which fails in a way that looks like a broken checkpoint. It isn't.

Use "method":"mtp". qwen3_next_mtp still works but logs a deprecation. When it loads correctly you'll see Resolved architecture: Qwen3_5MoeMTP in the startup log — that line is your confirmation the graft took.

Set --kv-cache-dtype fp8 at serve time. I could not bake kv_cache_scheme into the checkpoint: this model nests its text settings under text_config, so there is no top-level num_attention_heads and llm-compressor's KV path fails looking for it (ValueError: Cannot determine num_attention_heads from config). The flag does the same job at runtime.

This is a reasoning model and it writes its chain into content. At 512 max_tokens I got cut off mid-thought with no answer, which looks exactly like a broken quant. Use 1500+.

Benchmarks

One GB10, vLLM 0.26.1rc1, 64K context, FP8 KV, max_num_seqs 2, 256-token generations, nonce-prefixed prompts so nothing came from the prefix cache. Servers warmed before measuring. Medians of 3–4 runs.

	tok/s
NVFP4 + MTP	57.36
NVFP4, no MTP	42.49
Speedup	1.35×
MTP acceptance rate	73.9% (572 accepted / 774 drafted)

Both numbers are the same checkpoint on the same box with the same flags — the only difference is whether --speculative-config is passed. The no-MTP runs were very tight (42.3, 42.3, 42.6, 42.6), so this is a real delta, not noise.

Warm up before you measure. My first MTP reading was 47.2 tok/s and I nearly published 1.13× off it. That number was cold-start contamination; the steady-state figure is 57.4. If you benchmark this model on the first request after load you will underreport it.

Correctness after quantizing and after the graft: 17*23 → 391, capital of Japan → Tokyo, and a 256-token freeform explanation of mixture-of-experts that came back as clean coherent English. That last one matters more than it sounds — a badly quantized MoE loads fine and then emits repetition or token soup, so "it loaded" proves nothing.

No BF16 A/B. I did not benchmark the 67 GB original on the same box, so I'm not quoting a quality delta against it. That measurement doesn't exist here.

Vision

The vision tower is present and unquantized — 333 visual.* tensors deliberately excluded and still BF16 — so image input should work. I never exercised it. Everything I verified was text. Upstream is explicit that their own post-training and evaluations are text/code-only, so treat vision here as inherited, not validated.

How it was made

from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor import oneshot

recipe = QuantizationModifier(
    targets="Linear", scheme="NVFP4",
    ignore=[
        "re:.*lm_head",
        "re:visual.*", "re:model.visual.*",
        "re:.*mlp.gate$",          # the MoE ROUTER — not the experts' gate_proj
        "re:.*embed_tokens$",
        "re:.*shared_expert_gate$",
        "re:.*linear_attn.*",
        "re:.*router.*",
    ],
)
oneshot(model=model, recipe=recipe, dataset=calibration,   # 256 samples @ 2048 tokens
        pipeline="sequential", moe_calibrate_all_experts=True)

34 minutes on the GB10, quantized from the original BF16 weights, not requantized from a lower-bit source. Then the MTP graft on top.

The ignore list is the part that matters. Quantize the router (mlp.gate) or the embeddings and the model still loads clean and then talks nonsense — that's the classic MoE quantization failure and it's very easy to ship by accident. moe_calibrate_all_experts=True is also not optional: with 256 experts and 8 active per token, a calibration pass that doesn't force every expert to see data leaves most of them with garbage scales.

The model

Frontis-MA1-35B is FrontisAI's execution-grounded model for evolutionary machine-learning engineering, post-trained from Qwen3.6-35B-A3B. 35B total / ~3B active, 40 layers, 256 experts with 8 active per token, 262K context. It's trained around four program-transformation operators — Draft, Improve, Debug, Crossover — composed by an outer search loop.

Their headline result is MLE-Bench Lite Medal Average going from 39.39% to 60.61% over the base model under their OpenMLE-Evo harness. Those numbers measure a model-plus-harness system, not one-shot generation — reproducing them needs their harness and protocol, not just these weights.

Everything about behaviour, prompt format and capability comes from upstream. See the base model card and the paper. All I changed is the numeric format.

License

CC-BY-NC-4.0, inherited from the base model — this is non-commercial. Different from my other quants, which are Apache 2.0 or MIT. That license follows the weights and it follows this quant. Check it before you build on it.

Credit

The model is FrontisAI's; the NVFP4 format and kernels are NVIDIA's; the quantization tooling is llm-compressor from the vLLM project. I built the quant, grafted the MTP block, ran the benchmarks and wrote down what I found.