---
language:
- en
library_name: transformers
pipeline_tag: text-generation
datasets:
- epfml/FineWeb-HQ
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0
- HuggingFaceTB/smollm-corpus
- HuggingFaceTB/finemath
- AxiomicLabs/NPset-2-Python-Edu
tags:
- causal-lm
- base-model
- mixture-of-experts
- sequence-routing
- custom-code
- trust-remote-code
---

# SLMoE Test - 100%

This model has been released as https://huggingface.co/BananaMind/BananaMind-2-SLMoE

This is the **100% checkpoint** of an experimental sequence-routed base
language model. It is not instruction tuned.

## Architecture

| Field | Value |
|---|---:|
| Total parameters | 25,449,472 |
| Active parameters per sequence | 7,902,208 |
| Layers / hidden size | 8 / 256 |
| Query / KV heads | 8 / 2 |
| Head dimension | 32 |
| Expert paths | 64 total / 13 active |
| Expert intermediate size | 56 |
| Context | 4,096 |
| Vocabulary | 8,192, tied |

The router reads only the first 32 valid tokens and makes one global top-13
decision. Those expert identities are reused in every layer and stored in the
KV cache for the complete generated response. Routing does not change per
token. During pretraining, one packed 4,096-token sequence is the routing unit,
and loss on its routing prefix is masked to prevent future-token leakage.

## Training

| Field | Value |
|---|---:|
| Progress | 100% |
| Tokens seen | 59,999,518,720 |
| Target tokens | 60,000,000,000 |
| Hardware | 4 x NVIDIA H200 |
| Optimizer | Fused AdamW |
| Peak LR | 0.003 |
| Betas | 0.9, 0.95 |
| Weight decay | 0.1 |
| Precision | bfloat16 autocast |

| Token range | FineWeb-HQ | FineWeb-Edu | DCLM | Cosmopedia v2 | FineMath | NPSet-2 |
|---|---:|---:|---:|---:|---:|---:|
| 0.00B-12.00B | 35% | 30% | 20% | 10% | 4% | 1% |
| 12.00B-24.00B | 32% | 30% | 18% | 11% | 7% | 2% |
| 24.00B-39.00B | 28% | 29% | 16% | 13% | 11% | 3% |
| 39.00B-51.00B | 24% | 27% | 13% | 14% | 18% | 4% |
| 51.00B-60.00B | 20% | 24% | 9% | 15% | 26% | 6% |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Banaxi-Tech/slmoe-test"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
```