| --- |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| datasets: |
| - epfml/FineWeb-HQ |
| - HuggingFaceFW/fineweb-edu |
| - mlfoundations/dclm-baseline-1.0 |
| - HuggingFaceTB/smollm-corpus |
| - HuggingFaceTB/finemath |
| - AxiomicLabs/NPset-2-Python-Edu |
| tags: |
| - causal-lm |
| - base-model |
| - mixture-of-experts |
| - sequence-routing |
| - custom-code |
| - trust-remote-code |
| --- |
| |
| # SLMoE Test - 100% |
|
|
| This model has been released as https://huggingface.co/BananaMind/BananaMind-2-SLMoE |
|
|
| This is the **100% checkpoint** of an experimental sequence-routed base |
| language model. It is not instruction tuned. |
|
|
| ## Architecture |
|
|
| | Field | Value | |
| |---|---:| |
| | Total parameters | 25,449,472 | |
| | Active parameters per sequence | 7,902,208 | |
| | Layers / hidden size | 8 / 256 | |
| | Query / KV heads | 8 / 2 | |
| | Head dimension | 32 | |
| | Expert paths | 64 total / 13 active | |
| | Expert intermediate size | 56 | |
| | Context | 4,096 | |
| | Vocabulary | 8,192, tied | |
|
|
| The router reads only the first 32 valid tokens and makes one global top-13 |
| decision. Those expert identities are reused in every layer and stored in the |
| KV cache for the complete generated response. Routing does not change per |
| token. During pretraining, one packed 4,096-token sequence is the routing unit, |
| and loss on its routing prefix is masked to prevent future-token leakage. |
|
|
| ## Training |
|
|
| | Field | Value | |
| |---|---:| |
| | Progress | 100% | |
| | Tokens seen | 59,999,518,720 | |
| | Target tokens | 60,000,000,000 | |
| | Hardware | 4 x NVIDIA H200 | |
| | Optimizer | Fused AdamW | |
| | Peak LR | 0.003 | |
| | Betas | 0.9, 0.95 | |
| | Weight decay | 0.1 | |
| | Precision | bfloat16 autocast | |
|
|
| | Token range | FineWeb-HQ | FineWeb-Edu | DCLM | Cosmopedia v2 | FineMath | NPSet-2 | |
| |---|---:|---:|---:|---:|---:|---:| |
| | 0.00B-12.00B | 35% | 30% | 20% | 10% | 4% | 1% | |
| | 12.00B-24.00B | 32% | 30% | 18% | 11% | 7% | 2% | |
| | 24.00B-39.00B | 28% | 29% | 16% | 13% | 11% | 3% | |
| | 39.00B-51.00B | 24% | 27% | 13% | 14% | 18% | 4% | |
| | 51.00B-60.00B | 20% | 24% | 9% | 15% | 26% | 6% | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "Banaxi-Tech/slmoe-test" |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True) |
| ``` |
|
|
