| --- |
| language: en |
| license: cc-by-nc-4.0 |
| tags: |
| - complexity-deep |
| - token-routed |
| - moe |
| - deterministic-routing |
| - zipf-routing |
| - mu-guidance |
| library_name: complexity-framework |
| pipeline_tag: text-generation |
| --- |
| |
| # COMPLEXITY-DEEP Token-Routed MoE (383.5M) |
|
|
| ## Model Details |
|
|
| - **Architecture**: Token-Routed MLP + Mu-Guidance + Shared Lexical Expert |
| - **Parameters**: 383.5M total, ~105M active per token |
| - **Hidden size**: 1024 |
| - **Layers**: 20 |
| - **Attention heads**: 16 (GQA, 4 KV heads) |
| - **Intermediate size**: 3200 (800 per expert) |
| - **Experts**: 4 (deterministic Zipf-balanced routing) |
| - **Shared expert**: 800 intermediate |
| - **Vocabulary**: 32,000 |
| - **Max context**: 4,096 |
|
|
| ## Training |
|
|
| - **Dataset**: FineWeb-Edu (streaming) |
| - **Tokens**: 8B (15,259 steps) |
| - **Batch size**: 128 per GPU x 2 GPUs = 256 effective |
| - **Optimizer**: AdamW (lr=2.1e-4, auto-scaled to 4.2e-4) |
| - **Scheduler**: Cosine with 5% warmup (762 steps) |
| - **Precision**: BF16 |
| - **Hardware**: 2x NVIDIA RTX PRO 6000 (96GB each) |
| - **Training time**: ~30 hours |
|
|
| ## Results |
|
|
| ### Loss |
| - **Final loss**: ~2.96 |
| - **vs Dense baseline (384.5M)**: +0.09 gap (stable from step 5K) |
| - **Gap trend**: 0.28 (step 1K) -> 0.09 (step 5K+) |
|
|
| ### Zero-Shot Benchmarks |
| | Benchmark | MoE (383.5M) | Dense (384.5M) | |
| |-----------|-------------|----------------| |
| | ARC-Easy | 43.6% | 45.9% | |
| | HellaSwag | 28.7% | 30.1% | |
| | MMLU | 23.0% | 23.1% | |
|
|
| ### Inference (vLLM 0.18, RTX PRO 6000 96GB) |
| - **Sustained throughput**: 4,900 tok/s |
| - **Peak throughput**: 5,700 tok/s |
| - **Median TTFT**: 39.6 ms |
| - **Median ITL**: 16.0 ms |
| - **CUDA graph**: natively compatible (deterministic routing) |
|
|
| ## Expert Specialization (3D t-SNE) |
|
|
| Interactive visualization of expert activations across layers. Each point is an expert at a given layer; proximity = similar activation patterns. |
|
|
| **[▶ Open Interactive 3D t-SNE Visualization](https://www.complexity-ai.fr/expert_tsne_3d.html)** |
|
|
| ## Generation Example |
|
|
| No supervised fine-tuning. Raw base model output: |
|
|
| **Prompt:** "The meaning of life is" |
|
|
| **Output:** "very much the same. The same thing happens to all living things. They live in a constant |
| state of flux. The single cell of a living cell, in this case a cell nucleus, constantly changes to become |
| an organism, and that organism is the organism. The human body is a system of interconnected |
| cells. Each cell is made up of a set of parts, which are connected by a network of specialized cells." |
|
|
| ## Files |
|
|
| - `model.safetensors` - Model weights |
| - `model_config.yaml` - Architecture configuration |
| - `config.json` - HuggingFace-compatible config |
|
|
| ## Usage |
|
|
| ```python |
| from complexity.config import ModelConfig |
| from complexity.models import ComplexityModel |
| from safetensors.torch import load_file |
| |
| config = ModelConfig.load("model_config.yaml") |
| model = ComplexityModel(config) |
| state = load_file("model.safetensors", device="cpu") |
| model.load_state_dict(state, strict=False) |
| model.eval().cuda() |
| ``` |
|
|
| ## Paper |
|
|
| Under review at TMLR: https://openreview.net/forum?id=jZq6EVboC6 |
|
|
| ## License |
|
|
| CC-BY-NC-4.0 |
|
|
| Complexity-ML -- 2026 |
|
|
