Instructions to use Motif-Technologies/Motif-3-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Motif-Technologies/Motif-3-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Motif-Technologies/Motif-3-Base", trust_remote_code=True)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Motif-Technologies/Motif-3-Base", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Motif-Technologies/Motif-3-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Motif-Technologies/Motif-3-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Motif-Technologies/Motif-3-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Motif-Technologies/Motif-3-Base

SGLang

How to use Motif-Technologies/Motif-3-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Motif-Technologies/Motif-3-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Motif-Technologies/Motif-3-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Motif-Technologies/Motif-3-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Motif-Technologies/Motif-3-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Motif-Technologies/Motif-3-Base with Docker Model Runner:
```
docker model run hf.co/Motif-Technologies/Motif-3-Base
```

dongseokmotif commited on 15 days ago

Commit

a45182c

verified ·

1 Parent(s): 33ac2f6

Add base model card (README)

Browse files

Files changed (1) hide show

README.md +128 -0

README.md CHANGED Viewed

@@ -1,3 +1,131 @@
 ---
 license: mit
 ---

 ---
+library_name: transformers
+pipeline_tag: text-generation
+language:
+- en
+- ko
+tags:
+- motif
+- motif-3
+- mixture-of-experts
+- moe
+- multilingual
+- pretrained
+- base-model
 license: mit
 ---
+<div align="center">
+  <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/6836935d054aee793ffd78f1/L3Rw_g8vkvD8dqhOYGZhl.png" width="180" alt="Motif">
+</div>
+<hr>
+<div align="center" style="line-height: 1;">
+  <a href="https://motiftech.io" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Motif%20Technologies-1783ff?logoColor=white"/></a>
+  <a href="https://huggingface.co/Motif-Technologies" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Motif-ffc107?color=ffc107&logoColor=white"/></a>
+</div>
+<div align="center" style="line-height: 1;">
+  <img alt="Tech Report" src="https://img.shields.io/badge/📄%20Tech%20Report-To%20be%20updated-lightgrey"/>
+  <a href="https://huggingface.co/Motif-Technologies/Motif-3-Base/blob/main/LICENSE" target="_blank"><img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53"/></a>
+</div>
+<p align="center">
+📄&nbsp;&nbsp;<b>Motif 3: Technical Report</b> <sup>(to be updated)</sup>
+</p>
+## 1. Model Introduction
+**Motif 3 Base** is the **base pretrained checkpoint** of Motif 3 — a large-scale, decoder-only Mixture-of-Experts (MoE) language model with **314 billion total parameters** and **13.2 billion parameters activated per token**. It is built from the ground up by [Motif Technologies](https://motiftech.io) following a fully in-house, proprietary design — not a re-parameterization of an existing open-source architecture.
+This repository provides the **foundation model prior to post-training**: it has completed large-scale pretraining but has **not** undergone supervised fine-tuning, reinforcement learning, or preference/safety alignment. It is released for **further fine-tuning, continued pretraining, and research**. For the instruction-tuned, post-trained model, see **[Motif-Technologies/Motif-3](https://huggingface.co/Motif-Technologies/Motif-3)**.
+Motif 3 is built around **Grouped Differential Latent Attention (GDLA)**, which integrates grouped differential attention with the compressed key–value representation of Multi-head Latent Attention. The architecture further incorporates **modified manifold-constrained hyper-connections (mHC)**, **Expert-Specific PolyNorm** activations, and a **Multi-Token Prediction (MTP)** auxiliary objective to improve optimization stability, expert specialization, and training efficiency.
+The model is pretrained on approximately **12.5 trillion tokens** spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora, with additional emphasis on Korean, reasoning-intensive, legal, and financial data.
+### Key Features
+- 🧠 **Fine-grained sparse MoE** — 384 routed experts with only 8 activated per token (plus 1 shared expert), providing a large expert pool at limited per-token compute.
+- 📏 **Native 256K context** (262,144 tokens), trained with window-aware context parallelism.
+- ⚙️ **Novel architecture** — GDLA attention, Expert-Specific PolyNorm, and modified mHC, with MTP used as an auxiliary pretraining objective.
+- 🌐 **Multilingual & general-purpose**, with a strong bytes-per-token tokenizer for English, Korean, code, and math.
+- 🧩 **Clean foundation** — a base checkpoint intended as a starting point for supervised fine-tuning, RL, and domain adaptation.
+## 2. Model Summary
+<div align="center">
+<table>
+<tbody>
+<tr><td><b>Model Type</b></td><td>Base (pretrained, not instruction-tuned)</td></tr>
+<tr><td><b>Architecture</b></td><td>Mixture-of-Experts (MoE), decoder-only</td></tr>
+<tr><td><b>Total Parameters</b></td><td>~314B</td></tr>
+<tr><td><b>Activated Parameters</b></td><td>~13.2B / token</td></tr>
+<tr><td><b>Number of Layers</b></td><td>53 (2 dense + 51 MoE)</td></tr>
+<tr><td><b>Hidden Dimension</b></td><td>4096</td></tr>
+<tr><td><b>Dense FFN Intermediate</b></td><td>12,288 (first 2 layers)</td></tr>
+<tr><td><b>Attention</b></td><td>Grouped Differential Latent Attention (GDLA) with gated output</td></tr>
+<tr><td><b>Query / KV Heads</b></td><td>80 / 16</td></tr>
+<tr><td><b>Routed Experts</b></td><td>384 (top-8)</td></tr>
+<tr><td><b>Shared Experts</b></td><td>1</td></tr>
+<tr><td><b>Activation</b></td><td>Expert-Specific PolyNorm</td></tr>
+<tr><td><b>Residual</b></td><td>Modified manifold-constrained hyper-connections (mHC)</td></tr>
+<tr><td><b>Context Length</b></td><td>262,144 (256K)</td></tr>
+<tr><td><b>Vocabulary Size</b></td><td>220,160</td></tr>
+<tr><td><b>Pretraining Tokens</b></td><td>~12.5T</td></tr>
+<tr><td><b>Tensor Type</b></td><td>bfloat16</td></tr>
+</tbody>
+</table>
+</div>
+## 3. Intended Use & Limitations
+**Intended use.** Motif 3 Base is a foundation model. Typical uses are supervised fine-tuning (SFT), continued pretraining, reinforcement learning, distillation, and research on pretrained representations.
+**Not an assistant.** This checkpoint is **not instruction-tuned or aligned** and ships **without a chat template**. Do not expect it to follow instructions, hold a conversation, or refuse unsafe requests out of the box. Use it in **text-completion** mode (or fine-tune it first).
+**Limitations.** Because no alignment or safety tuning has been applied, outputs may be factually incorrect, biased, or otherwise unsafe. Downstream users are responsible for adding appropriate fine-tuning, evaluation, and safety mitigations before deployment.
+## 4. Evaluation Results
+We report the absolute performance of the pretrained base checkpoint under the prompting settings indicated below, to characterize the capabilities acquired during pretraining. We deliberately omit cross-model comparisons: base-model results are increasingly not published, and scores are highly sensitive to the evaluation harness and prompting protocol. Accuracy is reported for the multiple-choice and mathematics benchmarks; pass@1 is reported for HumanEval and MBPP. "CoT" denotes chain-of-thought prompting.
+<div align="center">
+| Benchmark | Setting | **Motif-3-Base** |
+|:---|:---:|:---:|
+| MMLU | 5-shot | 86.20 |
+| MMLU-Pro | 5-shot CoT | 68.56 |
+| ARC-C | 25-shot | 94.71 |
+| WinoGrande | 5-shot | 80.90 |
+| HellaSwag | 10-shot | 88.30 |
+| PIQA | 0-shot | 85.14 |
+| GSM8K | 8-shot CoT | 93.93 |
+| MATH | 4-shot CoT | 70.58 |
+| HumanEval | 0-shot | 73.70 |
+| MBPP | 3-shot | 84.60 |
+</div>
+## 5. Architecture
+Motif 3 is a fully in-house design and introduces several custom components (full details in the technical report, to be updated):
+- **Grouped Differential Latent Attention (GDLA)** — integrates grouped differential attention (asymmetric signal/noise heads with a token-dependent differential coefficient) with the compressed KV latent of Multi-head Latent Attention, plus a query-dependent output gate. Retains the expressive attention dynamics of differential attention while substantially reducing KV-cache requirements.
+- **Expert-Specific PolyNorm** — replaces the SiLU gate with a learned polynomial normalization whose coefficients are learned independently per expert, reducing activation outliers while allowing each expert to specialize.
+- **Modified manifold-constrained hyper-connections (mHC)** — replaces conventional residual additions with a doubly-stochastic (Birkhoff-polytope) mixing of 4 parallel residual streams; the post-mapping multiplier is annealed from 2 → 1 during pretraining to limit activation-outlier accumulation.
+- **Multi-Token Prediction (MTP)** — a DeepSeek-V3-style multi-token-prediction objective used as an auxiliary target during pretraining. (The MTP head is not shipped with this base checkpoint.)
+## 6. Access
+This model is **openly available** — anyone can download the weights, no access request required.
+## 7. License
+This model is released under the **MIT License**. See the [LICENSE](https://huggingface.co/Motif-Technologies/Motif-3-Base/blob/main/LICENSE) file for details.
+## 8. Citation
+_Citation information will be added upon release of the technical report._
+---
+© Motif Technologies. All rights reserved.