README.md

---
language: eo
license: mit
---

# EsperBERTo: A RoBERTa-like model for Esperanto

This is a RoBERTa-like model trained from scratch on the Esperanto language. 

## Model description

The model has 6 layers, 768 hidden size, 12 attention heads, and a total of 84 million parameters. It's based on the RoBERTa architecture. The tokenizer is a byte-level Byte-Pair Encoding (BPE) tokenizer trained from scratch on the same Esperanto corpus.

- **Model:** RoBERTa-like
- **Layers:** 6
- **Hidden size:** 768
- **Heads:** 12
- **Parameters:** 84M
- **Tokenizer:** Byte-level BPE
- **Vocabulary size:** 52,000

## Training data

The model was trained on the Esperanto portion of the OSCAR corpus (`oscar.eo.txt`), which is approximately 3GB in size.

## Training procedure

The model was trained for one epoch on the OSCAR corpus using the `Trainer` API from the `transformers` library. The training was performed on a single GPU.

### Hyperparameters
- `output_dir`: "./EsperBERTo"
- `overwrite_output_dir`: `True`
- `num_train_epochs`: 1
- `per_gpu_train_batch_size`: 64
- `save_steps`: 10_000
- `save_total_limit`: 2
- `prediction_loss_only`: `True`

The final training loss was `6.1178`.

## Evaluation results

The model was not evaluated on a downstream task in the notebook. However, its capabilities can be tested using the `fill-mask` pipeline.

Example 1:
```python
from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="./EsperBERTo",
    tokenizer="./EsperBERTo"
)

fill_mask("La suno <mask>.")
```
Output:
```
[{'score': 0.013023526407778263, 'token': 316, 'token_str': ' estas', 'sequence': 'La suno estas.'},
 {'score': 0.008523152209818363, 'token': 607, 'token_str': ' min', 'sequence': 'La suno min.'},
 {'score': 0.007405377924442291, 'token': 2575, 'token_str': ' okuloj', 'sequence': 'La suno okuloj.'},
 {'score': 0.007219308987259865, 'token': 1635, 'token_str': ' tago', 'sequence': 'La suno tago.'},
 {'score': 0.006888304837048054, 'token': 394, 'token_str': ' estis', 'sequence': 'La suno estis.'}]
```

Example 2:
```python
fill_mask("Jen la komenco de bela <mask>.")
```
Output:
```
[{'score': 0.016247423365712166, 'token': 1635, 'token_str': ' tago', 'sequence': 'Jen la komenco de bela tago.'},
 {'score': 0.009718689136207104, 'token': 1021, 'token_str': ' tempo', 'sequence': 'Jen la komenco de bela tempo.'},
 {'score': 0.007543196901679039, 'token': 2257, 'token_str': ' kongreso', 'sequence': 'Jen la komenco de bela kongreso.'},
 {'score': 0.0071307034231722355, 'token': 1161, 'token_str': ' vivo', 'sequence': 'Jen la komenco de bela vivo.'},
 {'score': 0.006644904613494873, 'token': 758, 'token_str': ' jaroj', 'sequence': 'Jen la komenco de bela jaroj.'}]
```

## Intended uses & limitations

This model is intended to be a general-purpose language model for Esperanto. It can be used for masked language modeling and can be fine-tuned for various downstream tasks such as:
- Text Classification
- Token Classification (Part-of-Speech Tagging, Named Entity Recognition)
- Question Answering

Since the model was trained on a relatively small dataset, its performance may be limited. For better results on specific tasks, fine-tuning on a relevant dataset is recommended.