| --- |
| language: |
| - multilingual |
| pipeline_tag: translation |
| tags: |
| - universal-translation |
| - nmt |
| - transformer |
| - encoder-decoder |
| - pytorch |
| license: apache-2.0 |
| datasets: |
| - code-with-zeeshan/UTS-Datasets |
| library_name: universal-translation-system |
| --- |
| |
| # Universal Translation System |
|
|
| A compact, production-ready multilingual neural machine translation model supporting **20 languages** (190 language pairs). Trained on curated OPUS-100 data with synthetic augmentation, knowledge distillation, and neural quality filtering. |
|
|
| ## Model Architecture |
|
|
| | Component | Configuration | |
| |-----------|--------------| |
| | Encoder | 6-layer Transformer, 512 hidden dim, 8 heads | |
| | Decoder | 8-layer Transformer, 768 hidden dim, 12 heads | |
| | Vocab | 32K tokens, script-grouped (latin, cjk, arabic, devanagari, cyrillic, thai) | |
| | Params | ~40MB (compact), ~150M total | |
| | Precision | BF16 mixed-precision training | |
|
|
| ## Supported Languages |
|
|
| | Group | Languages | |
| |-------|-----------| |
| | Latin | en, es, fr, de, it, pt, nl, sv, pl, id, vi, tr | |
| | CJK | zh, ja, ko | |
| | Arabic | ar | |
| | Devanagari | hi | |
| | Cyrillic | ru, uk | |
| | Thai | th | |
|
|
| ## Usage |
|
|
| ### Via the CLI (`uts`) |
|
|
| ```bash |
| # Translate a sentence |
| uts serve --config config/base.yaml |
| curl -X POST http://localhost:8000/translate \ |
| -H "Content-Type: application/json" \ |
| -d '{"text": "Hello world", "source": "en", "target": "es"}' |
| ``` |
|
|
| ### Via Python |
|
|
| ```python |
| from runtime.encoder.universal_encoder import UniversalEncoder |
| from runtime.cloud_decoder import OptimizedUniversalDecoder |
| |
| encoder = UniversalEncoder.from_pretrained("code-with-zeeshan/Universal-Translation-System") |
| decoder = OptimizedUniversalDecoder.from_pretrained("code-with-zeeshan/Universal-Translation-System") |
| # See docs/API.md for full inference examples |
| ``` |
|
|
| ### Via Hugging Face Hub |
|
|
| ```python |
| from transformers import AutoModelForSeq2SeqLM, AutoTokenizer |
| |
| model = AutoModelForSeq2SeqLM.from_pretrained("code-with-zeeshan/Universal-Translation-System") |
| tokenizer = AutoTokenizer.from_pretrained("code-with-zeeshan/Universal-Translation-System") |
| ``` |
|
|
| ## Training |
|
|
| The model was trained using the [Universal Translation System](https://github.com/code-with-zeeshan/universal-translation-system) pipeline: |
|
|
| 1. **Data pipeline** — OPUS-100 download, sampling, augmentation (false friends, idioms, backtranslation), COMET quality filtering |
| 2. **Knowledge distillation** — NLLB-3.3B teacher → compact student |
| 3. **Vocabulary** — Script-grouped SentencePiece tokenizer (32K per group) |
| 4. **Training** — BF16 mixed-precision, dynamic batch sizing, gradient checkpointing. ~10 epochs with cosine LR schedule. |
|
|
| ## Evaluation |
|
|
| | Metric | Score | |
| |--------|-------| |
| | BLEU (average across 190 pairs) | *Coming soon* | |
| | COMET (average) | *Coming soon* | |
|
|
| ## Files |
|
|
| - `encoder/` — Universal encoder weights |
| - `decoder/` — Optimized decoder weights |
| - `vocab/` — Script-grouped vocabulary packs |
| - `config.yaml` — Training configuration used for this model |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
