Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,99 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- multilingual
|
| 4 |
+
pipeline_tag: translation
|
| 5 |
+
tags:
|
| 6 |
+
- universal-translation
|
| 7 |
+
- nmt
|
| 8 |
+
- transformer
|
| 9 |
+
- encoder-decoder
|
| 10 |
+
- pytorch
|
| 11 |
license: apache-2.0
|
| 12 |
+
datasets:
|
| 13 |
+
- code-with-zeeshan/UTS-Datasets
|
| 14 |
+
library_name: universal-translation-system
|
| 15 |
---
|
| 16 |
+
|
| 17 |
+
# Universal Translation System
|
| 18 |
+
|
| 19 |
+
A compact, production-ready multilingual neural machine translation model supporting **20 languages** (190 language pairs). Trained on curated OPUS-100 data with synthetic augmentation, knowledge distillation, and neural quality filtering.
|
| 20 |
+
|
| 21 |
+
## Model Architecture
|
| 22 |
+
|
| 23 |
+
| Component | Configuration |
|
| 24 |
+
|-----------|--------------|
|
| 25 |
+
| Encoder | 6-layer Transformer, 512 hidden dim, 8 heads |
|
| 26 |
+
| Decoder | 8-layer Transformer, 768 hidden dim, 12 heads |
|
| 27 |
+
| Vocab | 32K tokens, script-grouped (latin, cjk, arabic, devanagari, cyrillic, thai) |
|
| 28 |
+
| Params | ~40MB (compact), ~150M total |
|
| 29 |
+
| Precision | BF16 mixed-precision training |
|
| 30 |
+
|
| 31 |
+
## Supported Languages
|
| 32 |
+
|
| 33 |
+
| Group | Languages |
|
| 34 |
+
|-------|-----------|
|
| 35 |
+
| Latin | en, es, fr, de, it, pt, nl, sv, pl, id, vi, tr |
|
| 36 |
+
| CJK | zh, ja, ko |
|
| 37 |
+
| Arabic | ar |
|
| 38 |
+
| Devanagari | hi |
|
| 39 |
+
| Cyrillic | ru, uk |
|
| 40 |
+
| Thai | th |
|
| 41 |
+
|
| 42 |
+
## Usage
|
| 43 |
+
|
| 44 |
+
### Via the CLI (`uts`)
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
# Translate a sentence
|
| 48 |
+
uts serve --config config/base.yaml
|
| 49 |
+
curl -X POST http://localhost:8000/translate \
|
| 50 |
+
-H "Content-Type: application/json" \
|
| 51 |
+
-d '{"text": "Hello world", "source": "en", "target": "es"}'
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
### Via Python
|
| 55 |
+
|
| 56 |
+
```python
|
| 57 |
+
from runtime.encoder.universal_encoder import UniversalEncoder
|
| 58 |
+
from runtime.cloud_decoder import OptimizedUniversalDecoder
|
| 59 |
+
|
| 60 |
+
encoder = UniversalEncoder.from_pretrained("code-with-zeeshan/Universal-Translation-System")
|
| 61 |
+
decoder = OptimizedUniversalDecoder.from_pretrained("code-with-zeeshan/Universal-Translation-System")
|
| 62 |
+
# See docs/API.md for full inference examples
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
### Via Hugging Face Hub
|
| 66 |
+
|
| 67 |
+
```python
|
| 68 |
+
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
|
| 69 |
+
|
| 70 |
+
model = AutoModelForSeq2SeqLM.from_pretrained("code-with-zeeshan/Universal-Translation-System")
|
| 71 |
+
tokenizer = AutoTokenizer.from_pretrained("code-with-zeeshan/Universal-Translation-System")
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
+
## Training
|
| 75 |
+
|
| 76 |
+
The model was trained using the [Universal Translation System](https://github.com/code-with-zeeshan/universal-translation-system) pipeline:
|
| 77 |
+
|
| 78 |
+
1. **Data pipeline** β OPUS-100 download, sampling, augmentation (false friends, idioms, backtranslation), COMET quality filtering
|
| 79 |
+
2. **Knowledge distillation** β NLLB-3.3B teacher β compact student
|
| 80 |
+
3. **Vocabulary** β Script-grouped SentencePiece tokenizer (32K per group)
|
| 81 |
+
4. **Training** β BF16 mixed-precision, dynamic batch sizing, gradient checkpointing. ~10 epochs with cosine LR schedule.
|
| 82 |
+
|
| 83 |
+
## Evaluation
|
| 84 |
+
|
| 85 |
+
| Metric | Score |
|
| 86 |
+
|--------|-------|
|
| 87 |
+
| BLEU (average across 190 pairs) | *Coming soon* |
|
| 88 |
+
| COMET (average) | *Coming soon* |
|
| 89 |
+
|
| 90 |
+
## Files
|
| 91 |
+
|
| 92 |
+
- `encoder/` β Universal encoder weights
|
| 93 |
+
- `decoder/` β Optimized decoder weights
|
| 94 |
+
- `vocab/` β Script-grouped vocabulary packs
|
| 95 |
+
- `config.yaml` β Training configuration used for this model
|
| 96 |
+
|
| 97 |
+
## License
|
| 98 |
+
|
| 99 |
+
Apache 2.0
|