| --- |
| library_name: transformers |
| license: apache-2.0 |
| pipeline_tag: translation |
| tags: |
| - translation |
| - multilingual |
| - moe |
| - mixture-of-experts |
| - fsdp |
| - seq2seq |
| - 8k-context |
| - document-packing |
| - pytorch |
| --- |
| |
| # Mythos-2.04B-MoE |
|
|
| > [!NOTE] |
| > This repository contains model weights and configuration files for the pre-trained model in the PyTorch and Hugging Face format. |
| > |
| > These artifacts are compatible with PyTorch, Hugging Face Transformers, vLLM, SGLang, TokenSpeed, etc. |
|
|
| > [!TIP] |
| > For users seeking managed, scalable translation inference without infrastructure maintenance, the official Mythos API service is in active development. In particular, **Mythos-2.04B-MoE** will be available as a high-throughput multilingual translation engine with production features, e.g., 8k context length by default, multi-stage translation from single sentences to full multi-page documents, and native 500+ language support. The service is coming soon. Stay tuned for updates. |
|
|
| Following the development of modern multilingual architectures, we are pleased to introduce **Mythos-2.04B-MoE**, the most capable generation in the Mythos open-model family to date. |
|
|
| Built on the architectural foundation of **Sparse Mixture-of-Experts (SMoE)**, Mythos-2.04B-MoE delivers substantial gains across translation quality, low-resource language understanding, zero-shot cross-lingual transfer, and long-horizon document translation. Mythos-2.04B-MoE brings these advances to a compact, deployment-friendly sparse model: a native multilingual translation model that understands 500+ global languages, designed to carry complex sentence, paragraph, and multi-page document translations through to completion with greater reliability. |
|
|
| --- |
|
|
| ## Mythos-2.04B-MoE Highlights |
|
|
| Mythos-2.04B-MoE features the following enhancements: |
|
|
| * **Core Translation Capabilities:** Comprehensive improvements across cross-lingual semantic alignment, high-resource language fluency, and zero-shot translation paths across low-resource dialects. |
| * **Sparse MoE Efficiency:** 8 specialized SwiGLU expert feedforwards per block with Top-2 dynamic routing, delivering 2.04B total parameter knowledge capacity while requiring only 678M active parameters per token. |
| * **Long-Horizon Context Handling:** Dedicated 8,192-token context window with Block-Diagonal Document Packing, effortlessly handling single sentences, multi-sentence paragraphs, and large documents without truncation. |
| * **Vocabulary & Script Coverage:** 128,000-token Byte-Level BPE subword vocabulary natively supporting all major writing systems (Latin, Cyrillic, Arabic, Devanagari, Hanzi, Kana, Hangul, African, and Indigenous scripts). |
| * **Downstream Compatibility:** Standard Encoder-Decoder interface compatible with popular inference harnesses and distributed frameworks. |
|
|
| --- |
|
|
| ## Model Overview |
|
|
| * **Type:** Encoder-Decoder Multilingual Translation Model with Sparse Mixture-of-Experts (SMoE) |
| * **Training Stage:** Pre-training |
| * **Language Model Architecture:** |
| * **Number of Parameters:** 2.04B (`2,037,643,264` Total Parameters) |
| * **Active Parameters / Token:** 678.7M (`678,688,768` Active Parameters) |
| * **Hidden Dimension:** 1,024 |
| * **Token Embedding:** 128,000 (Byte-Level BPE) |
| * **Number of Layers:** 24 Layers (12 Encoder Layers + 12 Decoder Layers) |
| * **Hidden Layout:** 24 × (GQA Attention $\to$ SwiGLU MoE FFN) |
| * **Gated Attention (GQA):** |
| * **Number of Attention Heads:** 16 for Q and 4 for KV (4× KV-Cache Compression) |
| * **Head Dimension:** 64 |
| * **Rotary Position Embedding (RoPE) Dimension:** 64 ($\theta = 100,000.0$) |
| * **Feed-Forward Network (Sparse MoE):** |
| * **Number of Experts:** 8 Experts per block |
| * **Routing Mechanism:** Top-2 Active Experts with Load Balancing + Router Z-Loss ($1\times 10^{-4}$) |
| * **Intermediate Dimension (per Expert):** 3,072 (SwiGLU) |
| * **LM Output:** 128,000 (Triton Liger-Fused Cross-Entropy) |
| * **Context Length:** 8,192 tokens natively with Block-Diagonal Document Packing |
|
|
| --- |
|
|
| ## Quickstart & Inference |
|
|
| ### 1. Installation |
| ```bash |
| pip install torch transformers tokenizers sacrebleu unbabel-comet |
| ``` |
|
|
| ### 2. Python Inference Code |
| ```python |
| import torch |
| from tokenizers import Tokenizer |
| |
| # Load Multilingual Tokenizer |
| tokenizer = Tokenizer.from_file("multilingual_tokenizer.json") |
| sos_id = tokenizer.token_to_id("[SOS]") |
| eos_id = tokenizer.token_to_id("[EOS]") |
| pad_id = tokenizer.token_to_id("[PAD]") |
| |
| # Source Text and Target Language Prompt |
| src_text = "Artificial intelligence and multilingual language models are transforming global human communication." |
| target_language = "fra" # ISO-639-3 Language Code (e.g., 'fra', 'deu', 'hin', 'ara', 'zho', 'spa', 'rus', 'jpn') |
| |
| prompt_text = f"<2{target_language}>" |
| enc_tokens = [sos_id] + tokenizer.encode(src_text).ids + [eos_id] |
| dec_tokens = [sos_id] + tokenizer.encode(prompt_text).ids |
| |
| encoder_input = torch.tensor([enc_tokens], dtype=torch.long) |
| decoder_input = torch.tensor([dec_tokens], dtype=torch.long) |
| |
| print(f"Source Text : {src_text}") |
| print(f"Target Lang : {target_language}") |
| print("Translating with Mythos-2.04B-MoE...") |
| ``` |
|
|
| --- |
|
|
| ## Training Methodology & Infrastructure |
|
|
| * **Pre-Training Dataset:** MaLA-LM Bilingual Translation Corpus (16.5 Billion tokens across 3,759 distributed Parquet shards covering 491+ unique languages and scripts). |
| * **Quality Filtering:** Strictly validated using **LASER Confidence Scores ($\ge 1.06$)** and **Bicleaner Filtering ($\ge 0.60$)** alongside length-ratio and Unicode Mojibake cleaning. |
| * **Round-Robin 500-Language Spooling:** Every on-disk chunk round-robins across **500 distinct language shards** sequentially, mixing all world languages into globally shuffled, balanced 8k sequences. |
| * **Distributed Compute:** 2× NVIDIA Tesla T4 GPUs with PyTorch **Fully Sharded Data Parallel (FSDP)**, FP16 mixed precision, and **Layer-Wise Reentrant Activation Checkpointing**. |
| * **Zero-Host RAM Footprint:** Streaming disk-spooler architecture keeping host CPU memory strictly below **`< 1.0 GB`** throughout training. |
|
|
| --- |
|
|
| ## Evaluation & Benchmarks |
|
|
| *(Benchmark results on the 500-Language Universal Validation Suite, COMET-22, BLEU, and ChrF++ will be published here upon completion of the pre-training run.)* |
|
|
| | Benchmark | Metric | Mythos-2.04B-MoE | Baseline | |
| | :--- | :---: | :---: | :---: | |
| | **Fixed 500-Language Suite** | Token Accuracy | *TBD* | — | |
| | **COMET-22** | Neural Semantic Score | *TBD* | — | |
| | **SacreBLEU** | Corpus BLEU | *TBD* | — | |
| | **ChrF++** | Character F-score | *TBD* | — | |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use **Mythos-2.04B-MoE** or its architecture in your research or applications, please cite: |
|
|
| ```bibtex |
| @misc{mythos2026multilingual, |
| author = {Adithyan AI}, |
| title = {Mythos-2.04B-MoE: A Sparse Mixture-of-Experts Large Language Model for Universal Multilingual Translation}, |
| year = {2026}, |
| publisher = {Hugging Face}, |
| howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos_translation}} |
| } |
| ``` |
|
|
