code-with-zeeshan commited on
Commit
2b3afaf
Β·
verified Β·
1 Parent(s): 2101cfd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +96 -0
README.md CHANGED
@@ -1,3 +1,99 @@
1
  ---
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - multilingual
4
+ pipeline_tag: translation
5
+ tags:
6
+ - universal-translation
7
+ - nmt
8
+ - transformer
9
+ - encoder-decoder
10
+ - pytorch
11
  license: apache-2.0
12
+ datasets:
13
+ - code-with-zeeshan/UTS-Datasets
14
+ library_name: universal-translation-system
15
  ---
16
+
17
+ # Universal Translation System
18
+
19
+ A compact, production-ready multilingual neural machine translation model supporting **20 languages** (190 language pairs). Trained on curated OPUS-100 data with synthetic augmentation, knowledge distillation, and neural quality filtering.
20
+
21
+ ## Model Architecture
22
+
23
+ | Component | Configuration |
24
+ |-----------|--------------|
25
+ | Encoder | 6-layer Transformer, 512 hidden dim, 8 heads |
26
+ | Decoder | 8-layer Transformer, 768 hidden dim, 12 heads |
27
+ | Vocab | 32K tokens, script-grouped (latin, cjk, arabic, devanagari, cyrillic, thai) |
28
+ | Params | ~40MB (compact), ~150M total |
29
+ | Precision | BF16 mixed-precision training |
30
+
31
+ ## Supported Languages
32
+
33
+ | Group | Languages |
34
+ |-------|-----------|
35
+ | Latin | en, es, fr, de, it, pt, nl, sv, pl, id, vi, tr |
36
+ | CJK | zh, ja, ko |
37
+ | Arabic | ar |
38
+ | Devanagari | hi |
39
+ | Cyrillic | ru, uk |
40
+ | Thai | th |
41
+
42
+ ## Usage
43
+
44
+ ### Via the CLI (`uts`)
45
+
46
+ ```bash
47
+ # Translate a sentence
48
+ uts serve --config config/base.yaml
49
+ curl -X POST http://localhost:8000/translate \
50
+ -H "Content-Type: application/json" \
51
+ -d '{"text": "Hello world", "source": "en", "target": "es"}'
52
+ ```
53
+
54
+ ### Via Python
55
+
56
+ ```python
57
+ from runtime.encoder.universal_encoder import UniversalEncoder
58
+ from runtime.cloud_decoder import OptimizedUniversalDecoder
59
+
60
+ encoder = UniversalEncoder.from_pretrained("code-with-zeeshan/Universal-Translation-System")
61
+ decoder = OptimizedUniversalDecoder.from_pretrained("code-with-zeeshan/Universal-Translation-System")
62
+ # See docs/API.md for full inference examples
63
+ ```
64
+
65
+ ### Via Hugging Face Hub
66
+
67
+ ```python
68
+ from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
69
+
70
+ model = AutoModelForSeq2SeqLM.from_pretrained("code-with-zeeshan/Universal-Translation-System")
71
+ tokenizer = AutoTokenizer.from_pretrained("code-with-zeeshan/Universal-Translation-System")
72
+ ```
73
+
74
+ ## Training
75
+
76
+ The model was trained using the [Universal Translation System](https://github.com/code-with-zeeshan/universal-translation-system) pipeline:
77
+
78
+ 1. **Data pipeline** β€” OPUS-100 download, sampling, augmentation (false friends, idioms, backtranslation), COMET quality filtering
79
+ 2. **Knowledge distillation** β€” NLLB-3.3B teacher β†’ compact student
80
+ 3. **Vocabulary** β€” Script-grouped SentencePiece tokenizer (32K per group)
81
+ 4. **Training** β€” BF16 mixed-precision, dynamic batch sizing, gradient checkpointing. ~10 epochs with cosine LR schedule.
82
+
83
+ ## Evaluation
84
+
85
+ | Metric | Score |
86
+ |--------|-------|
87
+ | BLEU (average across 190 pairs) | *Coming soon* |
88
+ | COMET (average) | *Coming soon* |
89
+
90
+ ## Files
91
+
92
+ - `encoder/` β€” Universal encoder weights
93
+ - `decoder/` β€” Optimized decoder weights
94
+ - `vocab/` β€” Script-grouped vocabulary packs
95
+ - `config.yaml` β€” Training configuration used for this model
96
+
97
+ ## License
98
+
99
+ Apache 2.0