|
|
--- |
|
|
library_name: transformers |
|
|
language: |
|
|
- mt |
|
|
license: cc-by-nc-sa-4.0 |
|
|
base_model: google/mt5-small |
|
|
datasets: |
|
|
- nlpaueb/multi_eurlex |
|
|
model-index: |
|
|
- name: mt5-small_multieurlex-mlt |
|
|
results: |
|
|
- task: |
|
|
type: text-classification |
|
|
name: Topic Classification |
|
|
dataset: |
|
|
type: multieurlex-mt |
|
|
name: nlpaueb/multi_eurlex |
|
|
config: mt |
|
|
metrics: |
|
|
- type: f1 |
|
|
args: macro |
|
|
value: 30.10 |
|
|
name: Macro-averaged F1 |
|
|
source: |
|
|
name: MELABench Leaderboard |
|
|
url: https://huggingface.co/spaces/MLRS/MELABench |
|
|
extra_gated_fields: |
|
|
Name: text |
|
|
Surname: text |
|
|
Date of Birth: date_picker |
|
|
Organisation: text |
|
|
Country: country |
|
|
I agree to use this model in accordance to the license and for non-commercial use ONLY: checkbox |
|
|
--- |
|
|
|
|
|
# mT5-Small (MultiEURLEX Maltese) |
|
|
|
|
|
This model is a fine-tuned version of [google/mt5-small](https://huggingface.co/google/mt5-small) on the [nlpaueb/multi_eurlex mt](https://huggingface.co/datasets/nlpaueb/multi_eurlex) dataset. |
|
|
It achieves the following results on the test set: |
|
|
- Loss: 0.3648 |
|
|
- F1: 0.3125 |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
|
|
The model is fine-tuned on a specific task and it should be used on the same or similar task. |
|
|
Any limitations present in the base model are inherited. |
|
|
|
|
|
## Training procedure |
|
|
|
|
|
The model was fine-tuned using a customised [script](https://github.com/MLRS/MELABench/blob/main/finetuning/run_seq2seq_classification.py). |
|
|
|
|
|
### Training hyperparameters |
|
|
|
|
|
The following hyperparameters were used during training: |
|
|
- learning_rate: 0.001 |
|
|
- train_batch_size: 32 |
|
|
- eval_batch_size: 8 |
|
|
- seed: 42 |
|
|
- optimizer: Use adafactor and the args are: |
|
|
No additional optimizer arguments |
|
|
- lr_scheduler_type: linear |
|
|
- num_epochs: 200.0 |
|
|
- early_stopping_patience: 20 |
|
|
|
|
|
### Training results |
|
|
|
|
|
| Training Loss | Epoch | Step | Validation Loss | F1 | |
|
|
|:-------------:|:-----:|:-----:|:---------------:|:------:| |
|
|
| 1.5559 | 1.0 | 548 | 0.4136 | 0.2994 | |
|
|
| 0.424 | 2.0 | 1096 | 0.3933 | 0.2995 | |
|
|
| 0.4078 | 3.0 | 1644 | 0.3755 | 0.3007 | |
|
|
| 0.3848 | 4.0 | 2192 | 0.3663 | 0.2990 | |
|
|
| 0.3714 | 5.0 | 2740 | 0.3571 | 0.2987 | |
|
|
| 0.3599 | 6.0 | 3288 | 0.3452 | 0.3010 | |
|
|
| 0.3436 | 7.0 | 3836 | 0.3237 | 0.3010 | |
|
|
| 0.3358 | 8.0 | 4384 | 0.3232 | 0.3009 | |
|
|
| 0.3292 | 9.0 | 4932 | 0.3145 | 0.2989 | |
|
|
| 0.3196 | 10.0 | 5480 | 0.3101 | 0.2983 | |
|
|
| 0.3045 | 11.0 | 6028 | 0.3111 | 0.2985 | |
|
|
| 0.301 | 12.0 | 6576 | 0.3009 | 0.2941 | |
|
|
| 0.3017 | 13.0 | 7124 | 0.3081 | 0.2911 | |
|
|
| 0.3008 | 14.0 | 7672 | 0.3077 | 0.2952 | |
|
|
| 0.2945 | 15.0 | 8220 | 0.3013 | 0.2982 | |
|
|
| 0.2933 | 16.0 | 8768 | 0.2941 | 0.2940 | |
|
|
| 0.2858 | 17.0 | 9316 | 0.3019 | 0.2918 | |
|
|
| 0.2849 | 18.0 | 9864 | 0.2933 | 0.2965 | |
|
|
| 0.2804 | 19.0 | 10412 | 0.2937 | 0.2918 | |
|
|
| 0.2814 | 20.0 | 10960 | 0.2969 | 0.2960 | |
|
|
| 0.2735 | 21.0 | 11508 | 0.2983 | 0.2925 | |
|
|
| 0.2735 | 22.0 | 12056 | 0.3021 | 0.2986 | |
|
|
| 0.2713 | 23.0 | 12604 | 0.2953 | 0.2956 | |
|
|
| 0.2704 | 24.0 | 13152 | 0.3007 | 0.2959 | |
|
|
| 0.2634 | 25.0 | 13700 | 0.3044 | 0.2986 | |
|
|
| 0.2678 | 26.0 | 14248 | 0.2996 | 0.3005 | |
|
|
| 0.2611 | 27.0 | 14796 | 0.2942 | 0.2961 | |
|
|
|
|
|
|
|
|
### Framework versions |
|
|
|
|
|
- Transformers 4.51.1 |
|
|
- Pytorch 2.7.0+cu126 |
|
|
- Datasets 3.2.0 |
|
|
- Tokenizers 0.21.1 |
|
|
|
|
|
## License |
|
|
|
|
|
This work is licensed under a |
|
|
[Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License][cc-by-nc-sa]. |
|
|
Permissions beyond the scope of this license may be available at [https://mlrs.research.um.edu.mt/](https://mlrs.research.um.edu.mt/). |
|
|
|
|
|
[![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa] |
|
|
|
|
|
[cc-by-nc-sa]: http://creativecommons.org/licenses/by-nc-sa/4.0/ |
|
|
[cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png |
|
|
|
|
|
## Citation |
|
|
|
|
|
This work was first presented in [MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP](https://arxiv.org/abs/2506.04385). |
|
|
Cite it as follows: |
|
|
|
|
|
```bibtex |
|
|
@inproceedings{micallef-borg-2025-melabenchv1, |
|
|
title = "{MELAB}enchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource {M}altese {NLP}", |
|
|
author = "Micallef, Kurt and |
|
|
Borg, Claudia", |
|
|
editor = "Che, Wanxiang and |
|
|
Nabende, Joyce and |
|
|
Shutova, Ekaterina and |
|
|
Pilehvar, Mohammad Taher", |
|
|
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025", |
|
|
month = jul, |
|
|
year = "2025", |
|
|
address = "Vienna, Austria", |
|
|
publisher = "Association for Computational Linguistics", |
|
|
url = "https://aclanthology.org/2025.findings-acl.1053/", |
|
|
doi = "10.18653/v1/2025.findings-acl.1053", |
|
|
pages = "20505--20527", |
|
|
ISBN = "979-8-89176-256-5", |
|
|
} |
|
|
``` |
|
|
|
