File size: 6,657 Bytes
70038b6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 | ---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/finemath
- HuggingFaceTB/smollm-corpus
language:
- en
pipeline_tag: text-generation
library_name: transformers
base_model: DedeProGames/DynamicMind-Mini
tags:
- causal-lm
- language-model
- base-model
- mixture-of-experts
- sparse-moe
- dynamicmind
- digit-tokenizer
- pytorch
- custom-code
- trust-remote-code
---
# DynamicMind-MoE
DynamicMind-MoE is a sparse mixture-of-experts decoder-only model trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) and [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath)
The model has **30.2M total parameters but activates only 8.9M per token**, and was initialised by sparse-upcycling [DynamicMind-Mini](https://huggingface.co/DedeProGames/DynamicMind-Mini). It keeps the same custom 8k-token byte-level BPE tokenizer with digit-aware tokenization, so inference cost per token is unchanged from the dense model while total capacity is 3.4x larger.
## Model Details
| Field | Value |
|---|---:|
| Total parameters | 30,150,912 |
| Active parameters per token | 8,917,248 |
| Architecture | Sparse MoE Llama-style decoder |
| Layers | 9 |
| Hidden size | 256 |
| Routed experts per layer | 14 |
| Shared experts per layer | 1 |
| Experts activated per token | 2 routed + 1 shared |
| Expert intermediate size | 256 |
| Attention heads | 8 |
| KV heads | 2 |
| Vocabulary size | 8,192 |
| Context length | 1,024 |
| Embeddings | Tied input/output embeddings |
| Weight format | safetensors |
## Mixture of Experts
Each block's dense MLP is replaced by one always-on shared expert plus 14 fine-grained routed experts, of which the router selects the top 2 per token. Shared + top-2 at intermediate size 256 reproduces the dense model's exact active parameter count.
Load balancing uses an auxiliary-loss-free bias (DeepSeek-V3 style): a per-expert bias steers *selection* toward idle experts while the combining weights come from the unbiased softmax, so balancing costs no gradient interference.
Routing is **per token, per layer** — a single sequence touches many different experts, and the 9 routers are independent of each other.
## Tokenizer
DynamicMind-MoE uses the same digit-aware 8k tokenizer as [DynamicMind-Mini](https://huggingface.co/DedeProGames/DynamicMind-Mini).
Digits are kept as separate tokens so numbers do not collapse into large number tokens during tokenization.
Digit IDs:
| Token | ID |
|---|---:|
| `1` | 9 |
| `2` | 10 |
| `3` | 11 |
| `4` | 12 |
| `5` | 13 |
| `6` | 14 |
| `7` | 15 |
| `8` | 16 |
| `9` | 17 |
| `0` | 18 |
## Training
| Field | Value |
|---|---:|
| Initialisation | Sparse upcycling from DynamicMind-Mini |
| Tokens seen | 10,008,133,632 |
| Optimizer steps | 8,484 |
| Sequence length | 1,024 |
| Tokens per optimizer step | 1,179,648 |
| Peak learning rate | 1e-4 |
| Min learning rate | 1e-5 |
| LR schedule | Cosine decay after 200 warmup steps |
| Optimizer | AdamW (betas 0.9, 0.95) |
| Weight decay | 0.1 |
| Gradient clipping | 1.0 |
| Seed | 1337 |
| Hardware | 1x RTX 3060 12GB, 39.5h |
Data mixture: FineWeb-Edu `sample-10BT` 50.0%, Cosmopedia-v2 27.8%, FineMath-4plus 22.2%.
Embeddings, attention and norms transferred 1:1 from the dense model. The dense MLP (intermediate size 768) splits exactly into three 256-wide slices — because `down_proj` sums over the intermediate axis — and each expert was seeded from one slice plus small noise to break router symmetry.
## Benchmarks
Self-reported results from the official [BananaMind Base Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1) script, all measured with the same runner, dtype (bfloat16) and GPU.
| Model | Total params | Active params | Overall Elo |
|---|---:|---:|---:|
| BananaMind-2-Medium | 55.9M | 55.9M | 1,037 |
| GPT-2 | 124M | 124M | 990 |
| BananaMind-2-Nano | 12.1M | 12.1M | 915 |
| **DynamicMind-MoE** | **30.2M** | **8.9M** | **912** |
| Pythia-14m-deduped | 14M | 14M | 909 |
| BananaMind-2-MoE | 26.1M | — | 903 |
| DynamicMind-Mini | 8.9M | 8.9M | 868 |
**Detailed DynamicMind-MoE result**
| Category | Passed | Elo |
|---|---:|---:|
| **Overall** | **137 / 350** | **912** |
| Language completion | 38 / 50 | 1,115 |
| Logical reasoning | 17 / 50 | 970 |
| World knowledge | 25 / 50 | 951 |
| Commonsense | 23 / 50 | 923 |
| Context tracking | 15 / 50 | 843 |
| Quantitative | 11 / 50 | 822 |
| Code completion | 8 / 50 | 793 |
Against the dense DynamicMind-Mini the MoE gains +44 Elo overall at identical inference cost, improving in six of seven categories. Quantitative is the exception (822 vs 837), despite FineMath being 22% of the corpus.
Scores are self-evaluated and may vary with the benchmark revision, Transformers version, dtype, hardware, and generation settings. Cross-tokenizer comparisons (e.g. against GPT-2's 50k vocabulary) carry a residual tokenizer effect that mean-per-token log-probability does not fully remove.
## Usage
This model uses custom architecture code, so load it with `trust_remote_code=True`.
Install dependencies:
```bash
pip install -U transformers safetensors torch
```
Run inference:
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "DedeProGames/DynamicMind-MoE"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16,
).cuda().eval()
prompt = "The meaning of life is "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
with torch.no_grad():
output = model.generate(
input_ids=input_ids,
max_new_tokens=64,
do_sample=False,
repetition_penalty=1.1,
pad_token_id=tokenizer.eos_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
## Limitations
This is a base model, not instruction-tuned — it continues text rather than following instructions. At 8.9M active parameters it reproduces register and structure well (encyclopedic text reads encyclopedic, code keeps valid indentation) but is frequently wrong on facts and arithmetic. Keep a finite generation limit and do not use it for high-stakes decisions.
## License
Apache 2.0
|