|
|
--- |
|
|
language: |
|
|
- eo |
|
|
- en |
|
|
- es |
|
|
- ca |
|
|
tags: |
|
|
- translation |
|
|
- machine-translation |
|
|
- marian |
|
|
- opus-mt |
|
|
- multilingual |
|
|
license: cc-by-4.0 |
|
|
pipeline_tag: translation |
|
|
metrics: |
|
|
- bleu |
|
|
- chrf |
|
|
--- |
|
|
|
|
|
# Catalan, English, Spanish -> Esperanto MT Model |
|
|
|
|
|
## Model description |
|
|
|
|
|
This repository contains a **multilingual MarianMT** model for **(English, Spanish, Catalan) → Esperanto** translation. |
|
|
|
|
|
## Usage |
|
|
|
|
|
The model is loaded and used with `transformers` as: |
|
|
|
|
|
```python |
|
|
from transformers import MarianMTModel, MarianTokenizer |
|
|
import torch |
|
|
|
|
|
model_name = "Helsinki-NLP/opus-mt-caenes-eo" |
|
|
|
|
|
device = "cuda" if torch.cuda.is_available() else "cpu" |
|
|
model = MarianMTModel.from_pretrained(model_name).to(device) |
|
|
tokenizer = MarianTokenizer.from_pretrained(model_name) |
|
|
|
|
|
source_texts = [ |
|
|
"Buenos días, qué tal?", |
|
|
"Bon dia, com estàs?", |
|
|
"Good morning, how are you?" |
|
|
] |
|
|
|
|
|
inputs = tokenizer(source_texts, return_tensors="pt", padding=True, truncation=True) |
|
|
inputs = {k: v.to(device) for k, v in inputs.items()} |
|
|
|
|
|
translated_ids = model.generate(inputs["input_ids"]) |
|
|
translated_texts = tokenizer.batch_decode(translated_ids, skip_special_tokens=True) |
|
|
|
|
|
for src, tgt in zip(source_texts, translated_texts): |
|
|
print(f"Source: {src} => Translated: {tgt}") |
|
|
```` |
|
|
|
|
|
## Training data |
|
|
|
|
|
The model was trained using **Tatoeba** parallel data, with **FLORES-200** used as the development set. |
|
|
|
|
|
Training sentence-pair counts: |
|
|
|
|
|
* **ca-eo**: 672,931 |
|
|
* **es-eo**: 4,677,945 |
|
|
* **eo-en**: 5,000,000 |
|
|
|
|
|
## Evaluation on FLORES |
|
|
|
|
|
| Language Pair | BLEU | ChrF++ | |
|
|
| ------------- | ----: | ----: | |
|
|
| spa-epo | 16.25 | 49.10 | |
|
|
| cat-epo | 21.43 | 51.37 | |
|
|
| eng-epo | 26.42 | 58.23 | |
|
|
|