| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3.5-35B-A3B-Base |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| datasets: |
| - open-thoughts/OpenThoughts3-1.2M |
| - open-r1/OpenR1-Math-220k |
| - HuggingFaceTB/smoltalk2 |
| - NousResearch/hermes-function-calling-v1 |
| - OpenAssistant/oasst2 |
| - FabricAI/maple |
| tags: |
| - fabric1.6 |
| - mixture-of-experts |
| - multimodal |
| - image-text-to-text |
| - video-text-to-text |
| - vision |
| - video |
| - reasoning |
| - agentic |
| - tool-calling |
| - long-context |
| - hybrid-attention |
| - transformers |
| - pytorch |
| - safetensors |
| - bf16 |
| model-index: |
| - name: Fabric 1.6 |
| results: |
| - task: |
| type: text-generation |
| name: Math and Reasoning |
| dataset: |
| type: aime-2025 |
| name: AIME25 |
| metrics: |
| - type: pass@1 |
| value: 92.8 |
| name: Pass@1 |
| - task: |
| type: text-generation |
| name: Math and Reasoning |
| dataset: |
| type: aime-2026 |
| name: AIME26 |
| metrics: |
| - type: pass@1 |
| value: 93.1 |
| name: Pass@1 |
| - task: |
| type: text-generation |
| name: Math and Reasoning |
| dataset: |
| type: hmmt-feb-2026 |
| name: HMMT26 |
| metrics: |
| - type: pass@1 |
| value: 83.2 |
| name: Pass@1 |
| - task: |
| type: text-generation |
| name: Math and Reasoning |
| dataset: |
| type: imo-answerbench |
| name: IMOAB |
| metrics: |
| - type: pass@1 |
| value: 79.2 |
| name: Pass@1 |
| - task: |
| type: text-generation |
| name: Math and Reasoning |
| dataset: |
| type: math-500 |
| name: M500 |
| metrics: |
| - type: accuracy |
| value: 84.8 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: gpqa |
| name: GPQA |
| metrics: |
| - type: accuracy |
| value: 86.7 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: gpqa-diamond |
| name: GPQA-D |
| metrics: |
| - type: accuracy |
| value: 84.9 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: hle |
| name: HLE |
| metrics: |
| - type: accuracy |
| value: 21.4 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: mmlu-pro |
| name: MMLU-P |
| metrics: |
| - type: accuracy |
| value: 85.6 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: mmlu-redux |
| name: MMLU-R |
| metrics: |
| - type: accuracy |
| value: 93.5 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Science and Knowledge |
| dataset: |
| type: ceval |
| name: C-Eval |
| metrics: |
| - type: accuracy |
| value: 92.3 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Code Generation |
| dataset: |
| type: livecodebench-v6 |
| name: LCB6 |
| metrics: |
| - type: pass@1 |
| value: 80.2 |
| name: Pass@1 |
| - task: |
| type: text-generation |
| name: Code Generation |
| dataset: |
| type: swe-bench-verified |
| name: SWEB-V |
| metrics: |
| - type: resolve-rate |
| value: 72.9 |
| name: Resolve Rate |
| - task: |
| type: text-generation |
| name: Code Generation |
| dataset: |
| type: swe-bench-pro |
| name: SWEB-P |
| metrics: |
| - type: resolve-rate |
| value: 50.1 |
| name: Resolve Rate |
| - task: |
| type: text-generation |
| name: Instruction Following |
| dataset: |
| type: ifeval |
| name: IFEval |
| metrics: |
| - type: instruction-level |
| value: 93.09 |
| name: Instruction Level |
| - task: |
| type: text-generation |
| name: General Reasoning |
| dataset: |
| type: gsm8k-platinum |
| name: GSM8K-Pt |
| metrics: |
| - type: accuracy |
| value: 95.73 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Agentic Tool Use |
| dataset: |
| type: tau3-bench |
| name: TAU3 |
| metrics: |
| - type: pass-rate |
| value: 67.2 |
| name: Pass Rate |
| - task: |
| type: image-text-to-text |
| name: Visual Question Answering |
| dataset: |
| type: mmmu-pro |
| name: MMMU-P |
| metrics: |
| - type: accuracy |
| value: 74.10 |
| name: Accuracy |
| - task: |
| type: image-text-to-text |
| name: Visual Question Answering |
| dataset: |
| type: realworldqa |
| name: RWQA |
| metrics: |
| - type: accuracy |
| value: 85.4 |
| name: Accuracy |
| - task: |
| type: text-generation |
| name: Agentic Tool Use |
| dataset: |
| type: mcp-atlas |
| name: MCP-A |
| metrics: |
| - type: completion |
| value: 62.8 |
| name: Completion |
| - task: |
| type: text-generation |
| name: Agentic Tool Use |
| dataset: |
| type: widesearch |
| name: WS |
| metrics: |
| - type: rubric-score |
| value: 60.3 |
| name: Rubric Score |
| - task: |
| type: image-text-to-text |
| name: Visual Question Answering |
| dataset: |
| type: mathvista-mini |
| name: MV-mini |
| metrics: |
| - type: accuracy |
| value: 86.6 |
| name: Accuracy |
| --- |
| |
| <div align="center"> |
| <picture> |
| <img src="banner_fabric1.6.png" width="100%" alt="Fabric AI"> |
| </picture> |
| </div> |
| |
| <hr> |
| <div align="center" style="line-height:1"> |
| <a href="https://huggingface.co/FabricAI" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-FabricAI-ffc107?color=e0a800&logoColor=white"/></a> |
| <a href="https://fabricai.co.uk" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Fabric%20AI-white?logo=globe&logoColor=white"/></a> |
| <a href="https://x.com/fabricai_uk" target="_blank"><img alt="X" src="https://img.shields.io/badge/X-%40fabricai_uk-white?logo=x&logoColor=black"/></a> |
| </div> |
|
|
| # Fabric 1.6 |
|
|
| **Fabric 1.6** is a 35-billion-parameter Mixture-of-Experts (MoE) reasoning model developed by **Fabric AI**, with approximately **3 billion parameters activated per token**. It is a native multimodal, agentic model built on a hybrid **Gated DeltaNet + Gated Attention** architecture, with explicit chain-of-thought reasoning, a native 262,144-token context window, and built-in Multi-Token Prediction (MTP) for up to 50% faster generation. |
|
|
| Fabric 1.6 is designed for agentic use in harnesses such as **OpenCode**, **Pi Agent**, **Hermes Agent** and other OpenAI-compatible tool-calling environments, and offers the option to preserve thinking context from past messages across long multi-turn sessions. |
|
|
| ## 1. Key Features |
|
|
| - **Hybrid Architecture**: Gated DeltaNet (linear attention) layers interleaved with Gated Attention layers inside a 256-expert MoE transformer — sub-quadratic scaling with full attention capacity where it matters. |
| - **Native Long Context**: 262,144 tokens natively, extensible up to **1,010,000 tokens**. |
| - **Multi-Token Prediction (MTP)**: predicts multiple future tokens per step for up to **50% faster generation**. |
| - **Native Multimodality**: accepts text, image and video inputs within the same model. |
| - **Explicit Reasoning**: produces an internal chain of thought before answering; reasoning is exposed in a structured format that can be streamed and stored. |
| - **Agentic by Design**: reliable structured tool-calling, long-horizon task execution, and preserved thinking context across turns. |
|
|
| ## 2. Model Summary |
|
|
| <div align="center"> |
| <table> |
| <tbody> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Architecture</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">Hybrid Gated DeltaNet + Gated Attention, Mixture-of-Experts (MoE)</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Total Parameters</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">35B</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Activated Parameters</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">~3B</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Number of Layers</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">40</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Layer Layout</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">10 × (3 × (Gated DeltaNet → MoE) → 1 × (Gated Attention → MoE))</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Hidden Dimension</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">2048</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Gated DeltaNet</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">32 value heads, 16 QK heads, head dimension 128</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Gated Attention</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">16 Q heads, 2 KV heads, head dimension 256, RoPE dim 64</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Number of Experts</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">256</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Selected Experts per Token</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">8 routed + 1 shared</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Expert Intermediate Dimension</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">512</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Vocabulary Size</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">248,320</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Context Length</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">262,144 (extensible to 1,010,000)</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Multi-Token Prediction</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">1 MTP layer (up to 50% faster generation)</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Vision Encoder</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">27-layer ViT, hidden 1152, patch 16, temporal patch 2</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Modality</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">Text, Image, Video</td> |
| </tr> |
| <tr> |
| <td align="center" style="vertical-align: middle; text-align: center"><strong>Precision</strong></td> |
| <td align="center" style="vertical-align: middle; text-align: center">BF16 (this repo)</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
|
|
| ## 3. Datasets Used to Train |
|
|
| Fabric 1.6 was developed from the [Qwen3.5-35B-A3B-Base](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base) foundation through **continuous pre-training** followed by **post-training** (supervised fine-tuning and reinforcement-learning-based alignment). |
|
|
| Pre-training was performed primarily on a large, in-house **proprietary synthetic dataset spanning code, mathematics and reasoning**, complemented by open reasoning corpora: |
|
|
| - [**OpenThoughts3-1.2M**](https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M) — 1.2M high-quality reasoning traces across mathematics, science, coding and general problem solving. |
| - [**OpenR1-Math-220k**](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k) — 225k mathematical problems with think-style solutions. |
|
|
| Post-training instruction data combines permissively licensed open corpora with proprietary data: |
|
|
| - [**smoltalk2**](https://huggingface.co/datasets/HuggingFaceTB/smoltalk2) — an Apache-2.0 SFT subset (~340k examples) covering multilingual instruction following, multi-turn reasoning, tool-calling traces, system chats and table understanding. |
| - [**hermes-function-calling-v1**](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) — structured tool-calling traces. |
| - [**oasst2**](https://huggingface.co/datasets/OpenAssistant/oasst2) — curated, reviewed conversational chains. |
| - [**maple**](https://huggingface.co/datasets/FabricAI/maple) — a proprietary instruction and reasoning corpus developed in-house by Fabric AI (CC-BY-4.0). |
|
|
| In total, approximately **12 billion tokens** were processed across the pre-training and post-training stages. Knowledge cutoff: July 2026. |
|
|
| ## 4. Evaluation Results |
|
|
| Fabric 1.6 was evaluated on 22 benchmarks with greedy decoding (temperature 0). |
|
|
| | Category | Benchmark | Score | |
| |---|---|---| |
| | **Math & Reasoning** | AIME25 | 92.8 | |
| | | AIME26 | 93.1 | |
| | | HMMT26 | 83.2 | |
| | | IMOAB | 79.2 | |
| | | M500 | 84.8 | |
| | **Science & Knowledge** | GPQA | 86.7 | |
| | | GPQA-D | 84.9 | |
| | | HLE | 21.4 | |
| | | MMLU-P | 85.6 | |
| | | MMLU-R | 93.5 | |
| | | C-Eval | 92.3 | |
| | **Coding** | LCB6 | 80.2 | |
| | | SWEB-V | 72.9 | |
| | | SWEB-P | 50.1 | |
| | | IFEval | 93.09 | |
| | **General Reasoning** | GSM8K-Pt | 95.73 | |
| | **Agentic Tools** | TAU3 | 67.2 | |
| | | MMMU-P | 74.10 | |
| | | RWQA | 85.4 | |
| | | MCP-A | 62.8 | |
| | | WS | 60.3 | |
| | | MV-mini | 86.6 | |
|
|
| ## 5. Deployment |
|
|
| > [!Note] |
| > Fabric 1.6 runs on the following inference engines with built-in MTP (multi-token prediction) support: |
|
|
| ### vLLM |
|
|
| Install: |
|
|
| ```bash |
| uv pip install vllm --torch-backend=auto |
| ``` |
|
|
| For deployment across an 8-GPU node (with built-in MTP support): |
|
|
| ```bash |
| vllm serve FabricAI/Fabric1.6 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' |
| ``` |
|
|
| For tool use: |
|
|
| ```bash |
| vllm serve FabricAI/Fabric1.6 --port 8000 --tensor-parallel-size 8 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder |
| ``` |
|
|
| ### SGLang |
|
|
| Install: |
|
|
| ```bash |
| uv pip install sglang[all] |
| ``` |
|
|
| For deployment across an 8-GPU node (with built-in MTP support): |
|
|
| ```bash |
| python -m sglang.launch_server --model-path FabricAI/Fabric1.6 --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 |
| ``` |
|
|
| For tool use: |
|
|
| ```bash |
| python -m sglang.launch_server --model-path FabricAI/Fabric1.6 --port 8000 --tp-size 8 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --tool-call-parser qwen3_coder |
| ``` |
|
|
| ## 6. Model Usage |
|
|
| Fabric 1.6 always has thinking enabled and returns `reasoning_content` alongside the answer. The model was trained in **preserved thinking history** mode: for multi-turn conversations and tool calls, pass the complete assistant message returned by the API back to `messages` as-is — including `reasoning_content` and `tool_calls`, not just `content` — so that reasoning from earlier turns remains available to later ones. |
|
|
| ## 7. License |
|
|
| The model weights are released under the [Apache License 2.0](LICENSE). |
|
|
| ## 8. Citation |
|
|
| If you use Fabric 1.6 in your work, please cite it as: |
|
|
| ```bibtex |
| @misc{fabric1.6, |
| title = {{Fabric1.6}: Agentic Open Model for Enterprises}, |
| url = {https://huggingface.co/FabricAI/Fabric1.6}, |
| author = {{Fabric AI}}, |
| month = {August}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## 9. Contact |
|
|
| For questions, collaborations or access requests, contact the Fabric AI research team at [research@fabricai.co.uk](mailto:research@fabricai.co.uk). |
|
|
