---

Motif-3-NVFP4

NVFP4-quantized checkpoint of Motif-Technologies/Motif-3. This repository hosts the NVFP4 build of Motif 3 for a reduced memory footprint on NVFP4-capable hardware. For the full model description, architecture, and evaluation, see the base model card.

1. Model Introduction

Motif 3 is a large-scale, decoder-only Mixture-of-Experts (MoE) language model with 314 billion total parameters and 13.2 billion parameters activated per token, built from the ground up by Motif Technologies following a fully in-house, proprietary design.

It is built around Grouped Differential Latent Attention (GDLA) and further incorporates modified manifold-constrained hyper-connections (mHC), Expert-Specific PolyNorm activations, and a Multi-Token Prediction (MTP) head.

This checkpoint (Motif-3-NVFP4) is an NVFP4-quantized version of the model weights, intended for efficient serving on NVIDIA Blackwell-class GPUs. The architecture, context length, and behavior otherwise match the base model.

Key Features

• 🗜️ NVFP4-quantized — lower memory footprint vs. the BF16 / Block-FP8 release.
• 🧠 Fine-grained sparse MoE — 384 routed experts with only 8 activated per token (plus 1 shared expert).
• 📏 Native 256K context (262,144 tokens).
• ⚙️ Novel architecture — GDLA attention, Expert-Specific PolyNorm, modified mHC, and a built-in MTP head enabling self-speculative decoding.
• 🌐 Multilingual & general-purpose, with particular strength on long-horizon agentic tool use.

2. Model Summary

Base Model	Motif-Technologies/Motif-3
Quantization	NVFP4 (weights)
Architecture	Mixture-of-Experts (MoE), decoder-only
Total Parameters	~314B
Activated Parameters	~13.2B / token
Number of Layers	53 (2 dense + 51 MoE)
Hidden Dimension	4096
Dense FFN Intermediate	12,288 (first 2 layers)
Attention	Grouped Differential Latent Attention (GDLA) with gated output
Query / KV Heads	80 / 16
Routed Experts	384 (top-8)
Shared Experts	1
Activation	Expert-Specific PolyNorm
Residual	Modified manifold-constrained hyper-connections (mHC)
MTP Head	1 layer (self-speculative decoding)
Context Length	262,144 (256K)
Vocabulary Size	220,160

Note on Parameter Count: This model uses packed 8-bit storage for 4-bit weights. The automatically calculated parameter count shown in the Hugging Face UI reports half the actual value. The true parameter count is ~314B.

3. Evaluation Results

Motif 3 performs particularly well on agentic and tool-oriented benchmarks (e.g. τ³-Banking, Terminal-Bench 2.1) while remaining competitive on coding, mathematical reasoning, and general knowledge.

For the full benchmark table and evaluation setup, please refer to the base model card.

3.1 NVFP4 Accuracy Benchmarks

To quantify the accuracy cost of quantization, we evaluate this NVFP4 checkpoint against the Block-FP8 release on the same suite, under identical prompts and decoding settings. Across the five benchmarks the two builds land within ~0.15 points of each other on average, so NVFP4 delivers its reduced memory footprint at essentially no accuracy penalty. Individual scores move in both directions (NVFP4 is ahead on AA-Omniscience accuracy and HLE, slightly behind on AA-LCR and non-hallucination), which is consistent with run-to-run evaluation noise rather than a systematic degradation.

Benchmark	NVFP4	Block-FP8
AA-Omniscience (public set) — accuracy	36.17	34.67
AA-Omniscience (public set) — non-hallucination	77.55	78.32
GPQA-Diamond	84.34	84.55
AA-LCR	70.50	72.30
HLE (25% random sampled)	39.70	39.15
Average	61.65	61.80

4. Architecture

Motif 3 introduces several custom components (full details in the technical report):

• Grouped Differential Latent Attention (GDLA) — integrates grouped differential attention with the compressed KV latent of Multi-head Latent Attention, plus a query-dependent output gate.
• Expert-Specific PolyNorm — a learned polynomial normalization with per-expert coefficients, reducing activation outliers while allowing each expert to specialize.
• Modified manifold-constrained hyper-connections (mHC) — doubly-stochastic mixing of 4 parallel residual streams, with the post-mapping multiplier annealed from 2 → 1 during pretraining.
• Multi-Token Prediction (MTP) — a 1-layer MTP head enabling self-speculative decoding at inference.

5. Deployment — vLLM (Recommended)

🖥️ Serving runs on just 2 × NVIDIA B200 GPUs

Thanks to NVFP4 quantization, this 314B checkpoint serves on two B200 GPUs on a single node — that is the validated deployment configuration for the command below.

• NVFP4 requires NVFP4-capable hardware (NVIDIA Blackwell / B200).
• The model ships with a built-in MTP head (num_nextn_predict_layers=1), so it supports self-speculative decoding — add --speculative-config as shown below (num_speculative_tokens: 1 is optimal for this model).
• If you encounter any issues, please open an HF issue.

• Docker image: ghcr.io/motiftechnologies/vllm:v0.20.2-motif3.rc3

B200 — 2 GPUs

vllm serve "Motif-Technologies/Motif-3-NVFP4" \
    --trust-remote-code \
    --quantization modelopt_nvfp4 \
    --speculative-config '{"model": "Motif-Technologies/Motif-3-NVFP4", "num_speculative_tokens": 1}' \
    --tensor-parallel-size 1 \
    --data-parallel-size 2 \
    --data-parallel-size-local 2 \
    --enable-expert-parallel \
    --enable-auto-tool-choice \
    --tool-call-parser motif \
    --reasoning-parser motif \
    --dtype bfloat16 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 262144 \
    --block-size 128 \
    --host 0.0.0.0 --port 8080

6. Access

This model is openly available — anyone can download the weights, no access request required.

7. License

This model is released under the MIT License. See the LICENSE file for details.

---

© Motif Technologies. All rights reserved.