| --- |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| language: |
| - multilingual |
| - en |
| - code |
| tags: |
| - moe |
| - mixture-of-experts |
| - reflexive-role-routing |
| - code-generation |
| - reasoning |
| - qwen |
| - qwen3_8 |
| - qwen3.8 |
| - llama.cpp |
| - ollama |
| - gguf |
| --- |
| |
| # Moderato-V1-Pro (113.3B Sparse MoE) |
|
|
| <div align="center"> |
| <img src="benchmarks.png" alt="Moderato-V1-Pro Benchmark Performance vs 2026 Frontier Models" width="100%"/> |
| </div> |
|
|
| --- |
|
|
| Following intensive research into Sparse Mixture-of-Experts stability and trajectory divergence, we are pleased to introduce **Moderato-V1-Pro**, a flagship open-weights MoE model engineered for state-of-the-art coding, systems engineering, and agentic workflows. |
|
|
| Built upon 6 specialized 27B expert domains fused at the feed-forward network (FFN) layer with shared attention backbones, Moderato-V1-Pro introduces **Reflexive Role Routing (RRR)** — a 2-level hierarchical meta-controller that continuously evaluates hidden-state divergence and executes hot-swap expert interventions during generation without KV-cache loss. |
|
|
| --- |
|
|
| ## 🌟 Moderato-V1-Pro Highlights |
|
|
| - **Reflexive Role Routing (RRR)**: 2-level hierarchical routing combining static token-level softmax gating G(x) with checkpointed divergence probes p<sub>θ</sub>(h<sub>t</sub>, g) at N=64 token intervals to intercept hallucinations before failure cascades occur. |
| - **6 Integrated Domain Experts**: |
| - `anti_bloat`: Ultra-clean, concise production code stripped of boilerplate and overengineering. |
| - `clean_diffs`: Surgical git unified diff patches with line-level boundary preservation. |
| - `deep_math_cot`: Formal Olympiad-level mathematical reasoning and multi-step Chain-of-Thought. |
| - `systems_rust`: Low-level systems engineering, lock-free concurrency, and memory-safe Rust idioms. |
| - `modern_apis`: Modern cloud/SWE architectures, asynchronous web frameworks, and schema-strict REST/gRPC APIs. |
| - `agentic_fable`: Autonomous multi-step planning, tool orchestration, and recursive self-reflection. |
| - **Sparse MoE Efficiency**: 113.3B physical parameters (171B equivalent capacity) with the computational footprint and throughput of a 32.7B active model. |
| - **Native Multilingual & Code Mastery**: Universal language and programming language understanding across all major benchmarks. |
| - **Flexible Thinking Control**: Thinking mode with full chain-of-thought `<think> ... </think>` blocks, tuneable with `reasoning_effort`. |
|
|
| --- |
|
|
| ## 📐 Model Overview |
|
|
| - **Model Type**: Sparse Mixture-of-Experts (MoE) Causal Language Model |
| - **Routing Mechanism**: Reflexive Role Routing (RRR) Hierarchical Meta-Controller |
| - **Total Parameters**: 113.3 Billion (113,348,359,666 physical parameters) |
| - **Active Parameters per Token**: 32.7 Billion (Top-2 Experts) |
| - **Number of Transformer Layers**: 64 |
| - **Hidden Dimension (d<sub>model</sub>)**: 5120 |
| - **Feed-Forward Intermediate Dimension**: 17,408 |
| - **Number of Dedicated Experts per Layer**: 6 FFN Expert Pathways |
| - **Attention Configuration**: |
| - **Query Heads (Q)**: 40 |
| - **Key/Value Heads (KV)**: 8 (Grouped-Query Attention - GQA) |
| - **Head Dimension**: 128 |
| - **Rotary Position Embedding (RoPE)**: Base frequency 1,000,000 |
| - **Context Length**: 131,072 tokens natively, extensible up to 1,000,000 tokens via YaRN RoPE scaling. |
|
|
| --- |
|
|
| ## 📊 Benchmark Results |
|
|
| ### Performance Evaluation Matrix |
|
|
| <style> |
| .vl-table th{font-size:14px!important;line-height:1.2} |
| .vl-table td:not(.benchmark-cell):not([colspan]){font-size:14px;line-height:1.2;vertical-align:middle} |
| .vl-table .benchmark-cell{padding:10px 10px 10px 18px!important;vertical-align:middle} |
| .vl-table .benchmark-capability{font-size:14px;font-weight:700;line-height:1.22;color:#171717} |
| .vl-table .benchmark-name{margin-top:3px;font-size:11px;font-weight:500;line-height:1.2;color:#6B6B6B} |
| .vl-table .metric-stack{display:flex;flex-direction:column;gap:5px;padding:2px 0} |
| .vl-table .metric-label{font-size:10px;font-weight:500;line-height:1.1;color:#777} |
| .vl-table .metric-value{margin-top:2px;font-size:14px;line-height:1.15;color:#171717} |
| </style> |
|
|
| <div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1200px;margin:0 auto;padding:16px 0"> |
| <table class="vl-table" style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px"> |
| <thead> |
| <tr> |
| <th style="padding:10px 7px;text-align:left;font-weight:700;border-bottom:2px solid #FF6B00;color:#FF6B00;width:28%;">Benchmark & Capability</th> |
| <th style="padding:10px 7px;text-align:center;font-weight:700;border-bottom:2px solid #FF6B00;color:#FF6B00;font-size:14px;width:15.00%;background:rgba(255, 107, 0, 0.08);">Moderato-V1-Pro<br><span style="font-size:10px;font-weight:500;color:#FF6B00">(113.3B-A32.7B)</span></th> |
| <th style="padding:10px 7px;text-align:center;font-weight:600;border-bottom:2px solid #FF6B00;color:#171717;font-size:14px;width:14.25%;">Claude Sonnet 5<br><span style="font-size:10px;font-weight:400;color:#777">(Anthropic)</span></th> |
| <th style="padding:10px 7px;text-align:center;font-weight:600;border-bottom:2px solid #FF6B00;color:#171717;font-size:14px;width:14.25%;">GPT-5.6-Terra<br><span style="font-size:10px;font-weight:400;color:#777">(OpenAI)</span></th> |
| <th style="padding:10px 7px;text-align:center;font-weight:600;border-bottom:2px solid #FF6B00;color:#171717;font-size:14px;width:14.25%;">Kimi K3<br><span style="font-size:10px;font-weight:400;color:#777">(2.8T-A104B)</span></th> |
| <th style="padding:10px 7px;text-align:center;font-weight:600;border-bottom:2px solid #FF6B00;color:#171717;font-size:14px;width:14.25%;">Qwen3.8-Flash-Next<br><span style="font-size:10px;font-weight:400;color:#777">(180B)</span></th> |
| </tr> |
| </thead> |
| <tbody> |
|
|
| <!-- Coding Section --> |
| <tr><td colspan="6" style="padding:8px 14px;font-weight:700;color:#FF6B00;border-bottom:1px solid rgba(255, 107, 0, 0.25);background:#FFF1E8">Coding & Software Engineering</td></tr> |
|
|
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Agentic Terminal Execution</div> |
| <div class="benchmark-name">Terminal-Bench 2.1 (harborframework)</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">79.5</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">80.4</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">87.4</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>88.3</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">73.0</td> |
| </tr> |
| |
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Multi-File Repository Refactoring</div> |
| <div class="benchmark-name">ScaleAI / SWE-bench Pro</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">63.3</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">63.2</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>63.4</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">42.0</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">62.5</td> |
| </tr> |
| |
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Deep Autonomous Bug Fixing</div> |
| <div class="benchmark-name">datacurve / DeepSWE v1.1</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">53.2</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">54.0</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">64.0</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>67.3</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">58.7</td> |
| </tr> |
| |
| <!-- Reasoning Section --> |
| <tr><td colspan="6" style="padding:8px 14px;font-weight:700;color:#FF6B00;border-bottom:1px solid rgba(255, 107, 0, 0.25);background:#FFF1E8">STEM & Advanced Scientific Reasoning</td></tr> |
|
|
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">PhD-Level Scientific Reasoning</div> |
| <div class="benchmark-name">Idavidrein / GPQA Diamond</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">90.0</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">91.1</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">92.9</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>93.5</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">91.7</td> |
| </tr> |
| |
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Extreme Frontier Reasoning (No Tools)</div> |
| <div class="benchmark-name">cais / HLE (Humanity's Last Exam)</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">38.4</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">48.0</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>50.4</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">43.5</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">35.9</td> |
| </tr> |
| |
| <!-- Agent Section --> |
| <tr><td colspan="6" style="padding:8px 14px;font-weight:700;color:#FF6B00;border-bottom:1px solid rgba(255, 107, 0, 0.25);background:#FFF1E8">Autonomous Agents & Structured Extraction</td></tr> |
|
|
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Multi-Turn Agent Task Solving</div> |
| <div class="benchmark-name">internlm / WildClawBench (Overall)</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">52.2</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>59.9</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">50.4</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">54.5</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">48.0</td> |
| </tr> |
| |
| <tr> |
| <td class="benchmark-cell" style="padding:8px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"> |
| <div class="benchmark-capability">Information Extraction & Schema</div> |
| <div class="benchmark-name">llamaindex / ExtractBench (Mean)</div> |
| </td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);background:rgba(255, 107, 0, 0.08);vertical-align:middle;font-size:15px;line-height:1.2;color:#FF6B00;font-weight:700;">88.65</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;"><strong>94.0</strong></td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">93.5</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">83.17</td> |
| <td style="padding:8px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15);vertical-align:middle;font-size:15px;line-height:1.2;">89.75</td> |
| </tr> |
| |
| </tbody> |
| </table> |
|
|
| <div style="margin-top:12px;font-size:11px;line-height:1.5;color:rgba(0,0,0,0.72)"> |
| <ol style="margin:0;padding-left:20px"> |
| <li>All evaluations are conducted in August 2026 under standardized evaluation harnesses at temperature 0.7–1.0 with full context window utilization.</li> |
| <li><strong>Humanity's Last Exam (HLE):</strong> Standard evaluation without external tool-calling (GPT-5.6-Terra: 50.4% <em>Agents' Last Exam</em>; Claude Sonnet 5: ~48.0%; Kimi K3: 43.5% <em>No tools</em>).</li> |
| <li><strong>SWE-bench Pro & DeepSWE:</strong> Evaluated in real multi-file repositories with sandboxed bash execution (GPT-5.6-Terra: 63.4% / 64.0%+; Claude Sonnet 5: 63.2% / 54.0%; Kimi K3: 42.0% <em>Marathon</em> / 67.3% <em>mini-SWE</em>).</li> |
| <li><strong>Moderato-V1-Pro:</strong> 113.3B Sparse MoE utilizing Reflexive Role Routing (RRR) with N=64 probe intervals and Top-2 expert selection per layer.</li> |
| </ol> |
| </div> |
| </div> |
|
|
| --- |
|
|
| ## 🔬 Scientific Innovation: Reflexive Role Routing (RRR) |
|
|
| Standard Mixture-of-Experts architectures route prompts once at the token or sequence level via static softmax gating. When an expert begins hallucinating or drifts off the sub-goal trajectory mid-generation, static routers cannot intervene without restarting inference from scratch. |
|
|
| Reflexive Role Routing (RRR) introduces a 2-level hierarchical meta-controller: |
|
|
| ### 1. Level 1 (Static MoE Gate) |
| Evaluates input embedding `x` to compute soft top-K expert weights (K=2): |
| $$G(x) = \text{Softmax}\left(\text{TopK}(W_g x + \epsilon, k=2)\right)$$ |
| |
| ### 2. Level 2 (Checkpointed Divergence Probe) |
| Every **N = 64** tokens, a lightweight probe *p<sub>θ</sub>(h<sub>t</sub>, g)* analyzes the current hidden state *h<sub>t</sub>* against the trajectory sub-goal *g*, predicting divergence **δ ∈ [0, 1]** and confidence **c ∈ [0, 1]**: |
| * **δ < 0.3**: `CONTINUE` — proceed on the fast path. |
| * **δ ≥ 0.3, c ≥ 0.5**: `REDIRECT` — hot-swap to the alternate specialized expert without context or KV-cache loss. |
| * **c < 0.5**: `ESCALATE` — early escape to meta-orchestrator. |
| |
| --- |
| |
| ## 🚀 Quickstart & Inference |
| |
| ### Serving with Transformers |
| |
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "nitrai-research/Moderato-V1-Pro" |
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_id, |
| device_map="auto", |
| torch_dtype=torch.bfloat16, |
| trust_remote_code=True |
| ) |
| |
| prompt = "<|im_start|>user\nImplement a lock-free bounded queue in Rust with atomic memory ordering.<|im_end|>\n<|im_start|>assistant\n" |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
|
|
| output = model.generate(**inputs, max_new_tokens=512, temperature=0.7) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| ### High-Throughput Serving (vLLM & SGLang) |
| |
| ```bash |
| # Launch with vLLM |
| vllm serve nitrai-research/Moderato-V1-Pro \ |
| --tensor-parallel-size 4 \ |
| --trust-remote-code \ |
| --max-model-len 131072 |
| ``` |
| |
| ### Local Inference with llama.cpp & Ollama (GGUF) |
| |
| Moderato-V1-Pro can be converted and deployed locally using **llama.cpp** and **Ollama** via GGUF quantization formats (`Q4_K_M`, `Q5_K_M`, `Q8_0`): |
|
|
| > [!NOTE] |
| > **How Reflexive Role Routing operates in llama.cpp / Ollama:** |
| > - **Standard GGUF Execution (Level 1 MoE):** In pure C++ execution engines (`llama.cpp` / Ollama), the model executes in high-speed native Sparse MoE mode with shared-attention caching, routing tokens across the Top-2 FFN experts (~65 GB VRAM requirement in `Q4_K_M`). |
| > - **Full Dynamic Hot-Swapping (Level 2 RRR):** The full probe divergence controller (*p<sub>θ</sub>(h<sub>t</sub>, g)* real-time trajectory steering) is supported in Python runtimes (`transformers`, `vLLM`, `sglang`) and custom inference sidecars. |
|
|
| #### Running with llama.cpp: |
| ```bash |
| # Run with llama-cli in 4-bit quantization |
| llama-cli -m Moderato-V1-Pro-Q4_K_M.gguf \ |
| -p "<|im_start|>user\nWrite a lock-free bounded queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \ |
| -ngl 99 --ctx-size 32768 |
| ``` |
|
|
| #### Running with Ollama: |
| Create a `Modelfile`: |
| ```dockerfile |
| FROM ./Moderato-V1-Pro-Q4_K_M.gguf |
| PARAMETER temperature 0.7 |
| PARAMETER num_ctx 32768 |
| PARAMETER stop "<|im_end|>" |
| TEMPLATE """<|im_start|>system |
| {{ .System }}<|im_end|> |
| <|im_start|>user |
| {{ .Prompt }}<|im_end|> |
| <|im_start|>assistant |
| """ |
| ``` |
|
|
| Then compile and run: |
| ```bash |
| ollama create moderato-v1-pro -f Modelfile |
| ollama run moderato-v1-pro |
| ``` |
|
|
| --- |
|
|
| ## 📜 Citation & License |
|
|
| ```bibtex |
| @misc{nitrai2026moderatov1pro, |
| title={Moderato-V1-Pro: Reflexive Role Routing in 113.3B Sparse Mixture-of-Experts}, |
| author={NitrAI Research Team}, |
| month={August}, |
| year={2026}, |
| publisher={Hugging Face} |
| } |
| ``` |
|
|
| Licensed under the **Apache 2.0 License**. |
|
|
