---
language:
- en
library_name: transformers
tags:
- moe
- mixture-of-experts
- small-model
- pretraining
- from-scratch
datasets:
- HuggingFaceFW/fineweb-edu
- HuggingFaceTB/cosmopedia-v2
- wikimedia/wikipedia
metrics:
- accuracy
---

# TinyMixtral v2

A 498M-parameter Mixture-of-Experts language model with a DeepSeek-style shared expert, trained from scratch on 4B tokens (FineWeb-Edu + Cosmopedia v2) and post-trained on 1B tokens (Wikipedia + Cosmopedia v2).

## Quick Start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral")

text = "The capital of France is"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Architecture

| Parameter | Value |
|-----------|-------|
| hidden_size | 896 |
| num_layers | 10 |
| Attention | GQA (16 heads / 4 KV heads, head_dim=56) |
| Experts | 1 shared (always active) + 6 routed (top-2) |
| Expert FFN | SwiGLU, intermediate = 2389 |
| Vocab size | 32,000 (TinyLlama SentencePiece) |
| Max position | 2,048 |
| Total params | ~498M |
| Active params/token | ~241M |

The shared expert provides general-purpose features and is always computed. Routed experts specialize via top-2 gating with load-balancing auxiliary loss.

## Training

| Phase | Data | Tokens | LR | Schedule | Time |
|-------|------|:------:|:---:|----------|:----:|
| Pretrain | FineWeb-Edu (89%) + Cosmopedia v2 (11%) | 4B | 7e-4 | WSD | ~96h |
| Post-train | Wikipedia (50%) + Cosmopedia v2 (50%) | 1B | 2e-5 | WSD | 24h |

- Hardware: single NVIDIA RTX A5000 24GB
- Precision: bf16 (model + autocast)
- Optimizer: AdamW (β=0.9,0.95, wd=0.1)
- Batch: 22 × 1024 = 22,528 tokens/step
- Gradient clipping: 1.0
- Activation checkpointing: enabled

## Results

### Standard Benchmarks (lm-evaluation-harness, 0-shot)

| Task | Metric | Score |
|------|--------|:---:|
| HellaSwag | acc_norm | 0.326 |
| PIQA | acc | 0.631 |
| WinoGrande | acc | 0.506 |
| ARC-Easy | acc | 0.474 |
| ARC-Challenge | acc_norm | 0.272 |
| OpenBookQA | acc_norm | 0.290 |
| BoolQ | acc | 0.455 |
| LAMBADA | acc | 0.224 |

Performance is comparable to SmolLM-360M on PIQA, ARC-C, and WinoGrande. HellaSwag and LAMBADA are weaker, reflecting limited next-token prediction capability at this scale.

### GLUE (zero-shot, conditional log-likelihood)

| Task | Score |
|------|:---:|
| SST2 | 0.586 |
| MRPC (F1) | 0.809 |
| QQP (F1) | 0.519 |
| QNLI | 0.458 |
| RTE | 0.520 |
| MNLI | 0.348 |
| MNLI-mm | 0.368 |
| **Mean** | **0.515** |

### Comparison with baseline (v1.1, 432M / 176M active)

| Metric | v1.1 | v2 (this model) |
|--------|:---:|:---:|
| GLUE Mean | 0.513 | **0.515** |
| ARC-C 0-shot | 0.249 | **0.264** |
| ARC-E 0-shot | 0.365 | **0.388** |
| SST2 | 0.568 | **0.586** |

v2 improves ARC (+1.5–2.3pp) and SST2 (+1.8pp) over v1.1, with GLUE mean roughly equivalent. The shared expert design increases knowledge capacity at the cost of slightly more parameters.

## Limitations

- At ~241M active parameters, performance on multi-step reasoning tasks (QNLI, MNLI, RTE) remains near random
- Trained on English data only
- Not instruction-tuned; best used as a base model or for text completion
- 2048 token context window

## Citation

```bibtex
@misc{tinymixtral2026,
  author = {Michael Lee},
  title = {TinyMixtral: Small-scale MoE Language Model Research},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/mikecovlee/tinymixtral}}
}
```

## License

MIT License. Copyright (C) 2026 Michael Lee (李登淳).