Add dense and top-2-of-4 MoE checkpoints, tokenizer, and benchmark assets

README.md

@@ -0,0 +1,207 @@
+---
+license: mit
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+inference: false
+datasets:
+- nampdn-ai/tiny-textbooks
+tags:
+- mixture-of-experts
+- moe
+- from-scratch
+- ablation
+- research
+---
+
+# MoE-Study — Dense vs. Mixture-of-Experts, matched active parameters
+
+Two decoder-only language models trained **from scratch** under identical conditions, differing in
+exactly one thing: whether the feed-forward block is a **dense MLP** or a **sparse top-2-of-4 MoE**.
+
+Both checkpoints live in this one repo:
+
+| Subfolder           | Model          | Total params | Active params/token |
+|---------------------|----------------|--------------|---------------------|
+| [`dense/`](./dense) | Dense FFN      | 150.1M       | 150.1M              |
+| [`moe/`](./moe)     | Top-2-of-4 MoE | 206.8M       | ~150.1M             |
+
+The MoE's active-parameter count matches Dense **by construction** — 2 of 4 experts at half the hidden
+size means identical compute per token. The MoE only spends more *memory* for extra capacity.
+
+Full write-up, training code, and evaluation harness:
+**[github.com/OliverSundaram/MoE-Study](https://github.com/OliverSundaram/MoE-Study)**
+
+---
+
+## ⚠️ These are research artifacts, not usable models
+
+Read this before downloading.
+
+- Trained for **one epoch** on ~40.7M tokens — neither model is close to converged.
+- **WikiText word perplexity is 551 (Dense) and 1,378 (MoE).** Generations are largely incoherent.
+- **0.0% on LAMBADA** for both — at the task floor.
+- No instruction tuning, no RLHF, no safety filtering of any kind.
+
+They exist to answer one narrow question: *at matched active compute and matched budget, does sparsity
+help?* They are not fit for any downstream use.
+
+---
+
+## Getting the weights
+
+These are a custom architecture, not a variant of an existing one. The modeling code is not included
+here, so `from_pretrained` on this repo alone will not build the model.
+
+Clone [the GitHub repo](https://github.com/OliverSundaram/MoE-Study) — it carries the model definition
+and loading instructions, and points back at these subfolders for the weights.
+
+---
+
+## Model details
+
+### Shared architecture
+
+Both models are the same custom decoder-only transformer:
+
+|                     |                                                                    |
+|---------------------|--------------------------------------------------------------------|
+| Layers              | 12                                                                 |
+| Attention heads     | 12                                                                 |
+| Embedding dim       | 768                                                                |
+| Context length      | 1024                                                               |
+| Vocabulary          | 50,257 (GPT-2 tokenizer)                                           |
+| Attention           | **Multi-Query** — one shared K/V projection across all query heads |
+| Normalization       | Custom pre-norm (learned scale + shift)                            |
+| Position embeddings | Learned absolute                                                   |
+| Weight tying        | None — separate input embedding and output head                    |
+
+### The one difference
+
+|              | `dense/`         | `moe/`                                         |
+|--------------|------------------|------------------------------------------------|
+| FFN block    | 2-layer GELU MLP | 4 experts, top-2 routed                        |
+| `hidden_dim` | 3072             | 1536 (per expert)                              |
+| Router       | —                | linear → softmax → top-2, renormalized         |
+| Aux loss     | —                | load-balancing term, summed over all 12 layers |
+
+Both models share the **same** unmodified GPT-2 tokenizer, stored once at the repo root.
+
+---
+
+## Training
+
+Identical for both models. Single consumer GPU, no cloud.
+
+| Setting       | Value                                                                                  |
+|---------------|----------------------------------------------------------------------------------------|
+| Data          | [`nampdn-ai/tiny-textbooks`](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks) |
+| Tokens        | 39,717 chunks × 1024 = **~40.67M**                                                     |
+| Epochs        | **1** (19,858 steps)                                                                   |
+| Batch size    | 2 × grad accum 4 = effective **8**                                                     |
+| Optimizer     | AdamW, lr `3e-4`, weight decay `0.1` (no decay on 1-D params)                          |
+| Schedule      | `OneCycleLR`, cosine, 3% warmup                                                        |
+| Grad clipping | max-norm `1.0`                                                                         |
+| Precision     | AMP autocast + `GradScaler`                                                            |
+| Seed          | 42                                                                                     |
+| Hardware      | 1× NVIDIA RTX 4060, 8 GB VRAM                                                          |
+| Wall-clock    | ~44.6 min (Dense) · ~59.8 min (MoE)                                                    |
+
+### Final losses
+
+|                            | Dense     | MoE       |
+|----------------------------|-----------|-----------|
+| Train loss (final step)    | 5.166     | 5.936     |
+| **Test loss (pure LM)**    | **5.063** | **5.911** |
+| Test loss (+ unscaled aux) | n/a       | 17.91     |
+
+Dense has the lower loss at **every** checkpoint.
+
+---
+
+## Evaluation
+
+All benchmarks via [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on the
+final checkpoints.
+
+| Benchmark        | Shots | Metric            | Dense     | MoE       | abs(Δ) | Winner   |
+|------------------|-------|-------------------|-----------|-----------|--------|----------|
+| ARC-Easy         | 0     | `acc`             | **29.2%** | 27.4%     | 1.8    | 🔵 Dense |
+| PIQA             | 0     | `acc`             | **55.0%** | 54.1%     | 0.9    | 🔵 Dense |
+| WikiText         | 0     | `word_perplexity` | **551.0** | 1,377.8   | 826.8  | 🔵 Dense |
+| LAMBADA (OpenAI) | 0     | `acc`             | 0.0%      | 0.0%      | 0.0    | ⚪ Tie    |
+| WinoGrande       | 5     | `acc`             | 50.2%     | **50.7%** | 0.5    | 🟠 MoE   |
+| HellaSwag        | 10    | `acc_norm`        | 24.9%     | **25.1%** | 0.2    | 🟠 MoE   |
+| ARC-Challenge    | 25    | `acc_norm`        | 22.9%     | **23.0%** | 0.1    | 🟠 MoE   |
+
+**How to read this:**
+
+- Dense wins on everything sensitive to raw LLM quality — perplexity, ARC-Easy, PIQA.
+- WinoGrande, HellaSwag, and ARC-Challenge are won by MoE, but with such a negligible difference, that they could be considered to have an equal accuracy
+
+### Inference speed
+
+Greedy decoding, 32-token prompt → 64 new tokens, 5 trials, 2 warmup, no KV cache.
+
+| Model | Tokens/sec        | Total params | Active params/token |
+|-------|-------------------|--------------|---------------------|
+| Dense | **106.49 ± 0.30** | 150.1M       | 150.1M              |
+| MoE   | 34.40 ± 0.08      | 206.8M       | ~150.1M             |
+
+MoE is **~3.1× slower** despite matched active compute — an artifact of unoptimized expert dispatch, not
+a property of the architecture.
+
+<details>
+<summary><b>Benchmark charts</b></summary>
+
+![ARC-Easy](./assets/arc_easy.png)
+![PIQA](./assets/piqa.png)
+![WikiText](./assets/wikitext.png)
+![LAMBADA](./assets/lambada_openai.png)
+![WinoGrande](./assets/winogrande.png)
+![HellaSwag](./assets/hellaswag.png)
+![ARC-Challenge](./assets/arc_challenge.png)
+![Speed](./assets/speed.png)
+</details>
+
+---
+
+## Findings
+
+**1. Dense won every metric that wasn't already at chance.**
+Most clearly on WikiText perplexity — 551 vs 1,378, a 2.5× gap.
+
+**2. The routing math is correct.**
+Active parameters match Dense almost exactly. Matched active compute simply didn't buy matched quality
+at this budget.
+
+**3. Routing stayed balanced.**
+The load-balancing term sat on its theoretical floor, so the MoE's gap is not explained by experts
+collapsing onto each other.
+
+**4. Extra capacity needs extra tokens.**
+The MoE has 38% more parameters but saw the same ~40.7M tokens — likely far too few to train 4 experts
+per layer, each seeing only a routed fraction of the stream.
+---
+
+## Citation
+
+```bibtex
+@misc{sundaram2026moestudy,
+  author = {Sundaram, Oliver},
+  title  = {MoE-Study: Dense vs. Mixture-of-Experts at Matched Active Parameters},
+  year   = {2026},
+  url    = {https://github.com/OliverSundaram/MoE-Study}
+}
+```
+
+## Acknowledgments
+
+- [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) (EleutherAI) — evaluation
+- [nampdn-ai/tiny-textbooks](https://huggingface.co/datasets/nampdn-ai/tiny-textbooks) — training corpus
+- [Hugging Face `transformers`](https://github.com/huggingface/transformers) — base classes and tokenizer
+
+## License
+
+MIT

dense/config.json

@@ -0,0 +1,18 @@
+{
+  "architectures": [
+    "LLM"
+  ],
+  "context_length": 1024,
+  "dtype": "float32",
+  "emb_dim": 768,
+  "hidden_dim": 3072,
+  "is_moe": false,
+  "model_type": "custom_llm",
+  "n_experts": null,
+  "n_heads": 12,
+  "n_layers": 12,
+  "qkv_bias": false,
+  "top_k": null,
+  "transformers_version": "5.15.0",
+  "vocab_size": 50257
+}

dense/final_state.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac21bb0a177dbeea395f7fc451c6dc10b079ddebaa971ab4ff976eac00ecbf4b
+size 1201509219

dense/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f1143cf1b87fa9e4cc8504f31a87058889922e4836e3907bd028e8db56ef2ab0
+size 600351628

moe/config.json

@@ -0,0 +1,18 @@
+{
+  "architectures": [
+    "LLM"
+  ],
+  "context_length": 1024,
+  "dtype": "float32",
+  "emb_dim": 768,
+  "hidden_dim": 1536,
+  "is_moe": true,
+  "model_type": "custom_llm",
+  "n_experts": 4,
+  "n_heads": 12,
+  "n_layers": 12,
+  "qkv_bias": false,
+  "top_k": 2,
+  "transformers_version": "5.15.0",
+  "vocab_size": 50257
+}

moe/final_state.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8107ca89b5111e58627130c68161136a9000ed0b342a1593ea4db05d8d7b9c00
+size 1655436555

moe/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de50a32d2c285961cd36bd51a5f4fe7e5593897318d31ab665f4d1573c1854d2
+size 827267812

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1024,
+  "pad_token": null,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}