| --- |
| language: |
| - en |
| library_name: transformers |
| tags: |
| - moe |
| - mixture-of-experts |
| - small-model |
| - pretraining |
| - from-scratch |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| - HuggingFaceTB/cosmopedia-v2 |
| - wikimedia/wikipedia |
| metrics: |
| - accuracy |
| --- |
| |
| # TinyMixtral v2 |
|
|
| A 498M-parameter Mixture-of-Experts language model with a DeepSeek-style shared expert, trained from scratch on 4B tokens (FineWeb-Edu + Cosmopedia v2) and post-trained on 1B tokens (Wikipedia + Cosmopedia v2). |
|
|
| ## Quick Start |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("mikecovlee/tinymixtral", trust_remote_code=True) |
| tokenizer = AutoTokenizer.from_pretrained("mikecovlee/tinymixtral") |
| |
| text = "The capital of France is" |
| inputs = tokenizer(text, return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.7) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
| ``` |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | hidden_size | 896 | |
| | num_layers | 10 | |
| | Attention | GQA (16 heads / 4 KV heads, head_dim=56) | |
| | Experts | 1 shared (always active) + 6 routed (top-2) | |
| | Expert FFN | SwiGLU, intermediate = 2389 | |
| | Vocab size | 32,000 (TinyLlama SentencePiece) | |
| | Max position | 2,048 | |
| | Total params | ~498M | |
| | Active params/token | ~241M | |
| |
| The shared expert provides general-purpose features and is always computed. Routed experts specialize via top-2 gating with load-balancing auxiliary loss. |
| |
| ## Training |
| |
| | Phase | Data | Tokens | LR | Schedule | Time | |
| |-------|------|:------:|:---:|----------|:----:| |
| | Pretrain | FineWeb-Edu (89%) + Cosmopedia v2 (11%) | 4B | 7e-4 | WSD | ~96h | |
| | Post-train | Wikipedia (50%) + Cosmopedia v2 (50%) | 1B | 2e-5 | WSD | 24h | |
| |
| - Hardware: single NVIDIA RTX A5000 24GB |
| - Precision: bf16 (model + autocast) |
| - Optimizer: AdamW (β=0.9,0.95, wd=0.1) |
| - Batch: 22 × 1024 = 22,528 tokens/step |
| - Gradient clipping: 1.0 |
| - Activation checkpointing: enabled |
| |
| ## Results |
| |
| ### Standard Benchmarks (lm-evaluation-harness, 0-shot) |
| |
| | Task | Metric | Score | |
| |------|--------|:---:| |
| | HellaSwag | acc_norm | 0.326 | |
| | PIQA | acc | 0.631 | |
| | WinoGrande | acc | 0.506 | |
| | ARC-Easy | acc | 0.474 | |
| | ARC-Challenge | acc_norm | 0.272 | |
| | OpenBookQA | acc_norm | 0.290 | |
| | BoolQ | acc | 0.455 | |
| | LAMBADA | acc | 0.224 | |
|
|
| Performance is comparable to SmolLM-360M on PIQA, ARC-C, and WinoGrande. HellaSwag and LAMBADA are weaker, reflecting limited next-token prediction capability at this scale. |
|
|
| ### GLUE (zero-shot, conditional log-likelihood) |
|
|
| | Task | Score | |
| |------|:---:| |
| | SST2 | 0.586 | |
| | MRPC (F1) | 0.809 | |
| | QQP (F1) | 0.519 | |
| | QNLI | 0.458 | |
| | RTE | 0.520 | |
| | MNLI | 0.348 | |
| | MNLI-mm | 0.368 | |
| | **Mean** | **0.515** | |
|
|
| ### Comparison with baseline (v1.1, 432M / 176M active) |
|
|
| | Metric | v1.1 | v2 (this model) | |
| |--------|:---:|:---:| |
| | GLUE Mean | 0.513 | **0.515** | |
| | ARC-C 0-shot | 0.249 | **0.264** | |
| | ARC-E 0-shot | 0.365 | **0.388** | |
| | SST2 | 0.568 | **0.586** | |
|
|
| v2 improves ARC (+1.5–2.3pp) and SST2 (+1.8pp) over v1.1, with GLUE mean roughly equivalent. The shared expert design increases knowledge capacity at the cost of slightly more parameters. |
|
|
| ## Limitations |
|
|
| - At ~241M active parameters, performance on multi-step reasoning tasks (QNLI, MNLI, RTE) remains near random |
| - Trained on English data only |
| - Not instruction-tuned; best used as a base model or for text completion |
| - 2048 token context window |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{tinymixtral2026, |
| author = {Michael Lee}, |
| title = {TinyMixtral: Small-scale MoE Language Model Research}, |
| year = {2026}, |
| publisher = {GitHub}, |
| howpublished = {\url{https://github.com/mikecovlee/tinymixtral}} |
| } |
| ``` |
|
|
| ## License |
|
|
| MIT License. Copyright (C) 2026 Michael Lee (李登淳). |
|
|
