| --- |
| library_name: transformers |
| language: |
| - mt |
| license: cc-by-nc-sa-4.0 |
| base_model: MLRS/BERTu |
| datasets: |
| - MLRS/SentiMalti |
| model-index: |
| - name: BERTu_SentiMalti |
| results: |
| - task: |
| type: sentiment-analysis |
| name: Sentiment Analysis |
| dataset: |
| type: sentimalti |
| name: SentiMalti |
| metrics: |
| - type: f1 |
| args: macro |
| value: 69.07 |
| name: Macro-averaged F1 |
| extra_gated_fields: |
| Name: text |
| Surname: text |
| Date of Birth: date_picker |
| Organisation: text |
| Country: country |
| I agree to use this model in accordance to the license and for non-commercial use ONLY: checkbox |
| --- |
| |
| # BERTu (Maltese Sentiment Analysis) |
|
|
| <img src="https://raw.githubusercontent.com/MLRS/BERTu/master/logo.png" width="200" margin-right="1em" align="left" /> |
|
|
| This model is a fine-tuned version of [MLRS/BERTu](https://huggingface.co/MLRS/BERTu) on [SentiMalti](https://huggingface.co/datasets/MLRS/SentiMalti). |
| It achieves the following results on the test set: |
| - Loss: 1.9082 |
| - F1: 0.6907 |
|
|
| ## Intended uses & limitations |
|
|
| The model is fine-tuned on a specific task and it should be used on the same or similar task. |
| Any limitations present in the base model are inherited. |
|
|
| ## Training procedure |
|
|
| The model was fine-tuned using a customised [script](https://github.com/MLRS/MELABench/blob/main/finetuning/run_classification.py). |
|
|
| ### Training hyperparameters |
|
|
| The following hyperparameters were used during training: |
| - learning_rate: 2e-05 |
| - train_batch_size: 16 |
| - eval_batch_size: 32 |
| - seed: 3 |
| - optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments |
| - lr_scheduler_type: inverse_sqrt |
| - lr_scheduler_warmup_ratio: 0.005 |
| - num_epochs: 200.0 |
| - early_stopping_patience: 20 |
|
|
| ### Training results |
|
|
| | Training Loss | Epoch | Step | Validation Loss | F1 | |
| |:-------------:|:-----:|:----:|:---------------:|:------:| |
| | 0.9723 | 1.0 | 165 | 0.7271 | 0.6184 | |
| | 0.6386 | 2.0 | 330 | 0.6306 | 0.6998 | |
| | 0.3595 | 3.0 | 495 | 0.7487 | 0.6893 | |
| | 0.172 | 4.0 | 660 | 1.0045 | 0.7018 | |
| | 0.0827 | 5.0 | 825 | 1.2351 | 0.6827 | |
| | 0.035 | 6.0 | 990 | 1.3912 | 0.7184 | |
| | 0.0195 | 7.0 | 1155 | 1.5390 | 0.7162 | |
| | 0.0078 | 8.0 | 1320 | 1.7268 | 0.7061 | |
| | 0.0038 | 9.0 | 1485 | 1.7315 | 0.7188 | |
| | 0.0027 | 10.0 | 1650 | 1.7508 | 0.7153 | |
| | 0.0036 | 11.0 | 1815 | 1.8214 | 0.7069 | |
| | 0.0004 | 12.0 | 1980 | 1.8759 | 0.7163 | |
| | 0.0004 | 13.0 | 2145 | 1.9154 | 0.7058 | |
| | 0.0004 | 14.0 | 2310 | 1.9490 | 0.7072 | |
| | 0.0002 | 15.0 | 2475 | 1.9955 | 0.7116 | |
| | 0.0002 | 16.0 | 2640 | 2.0237 | 0.7116 | |
| | 0.0003 | 17.0 | 2805 | 2.0499 | 0.6976 | |
| | 0.0002 | 18.0 | 2970 | 2.0582 | 0.7071 | |
| | 0.0013 | 19.0 | 3135 | 2.1837 | 0.6997 | |
| | 0.0003 | 20.0 | 3300 | 2.1333 | 0.7078 | |
| | 0.0002 | 21.0 | 3465 | 2.1226 | 0.7131 | |
| | 0.0021 | 22.0 | 3630 | 2.1613 | 0.7085 | |
| | 0.0001 | 23.0 | 3795 | 2.2053 | 0.7116 | |
| | 0.0001 | 24.0 | 3960 | 2.2182 | 0.7154 | |
| | 0.0013 | 25.0 | 4125 | 2.1939 | 0.7119 | |
| | 0.0001 | 26.0 | 4290 | 2.2181 | 0.7134 | |
| | 0.0001 | 27.0 | 4455 | 2.2612 | 0.7185 | |
| | 0.0018 | 28.0 | 4620 | 2.2793 | 0.7120 | |
| | 0.0001 | 29.0 | 4785 | 2.3041 | 0.7123 | |
|
|
|
|
| ### Framework versions |
|
|
| - Transformers 4.51.1 |
| - Pytorch 2.8.0+cu128 |
| - Datasets 3.2.0 |
| - Tokenizers 0.21.4 |
|
|
| ## License |
|
|
| This work is licensed under a |
| [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License][cc-by-nc-sa]. |
| Permissions beyond the scope of this license may be available at [https://mlrs.research.um.edu.mt/](https://mlrs.research.um.edu.mt/). |
|
|
| [![CC BY-NC-SA 4.0][cc-by-nc-sa-image]][cc-by-nc-sa] |
|
|
| [cc-by-nc-sa]: http://creativecommons.org/licenses/by-nc-sa/4.0/ |
| [cc-by-nc-sa-image]: https://licensebuttons.net/l/by-nc-sa/4.0/88x31.png |
|
|
| ## Citation |
|
|
| This work was first presented in [SentiMalti: A Maltese Sentiment Analysis Dataset and Models](#). |
| Cite it as follows: |
|
|
| ```bibtex |
| @inproceedings{SentiMalti, |
| title = "{S}enti{M}alti: A {M}altese Sentiment Analysis Dataset and Models", |
| author = "Caruana, Ian and |
| Vella, Matthew and |
| Zammit, Fabio and |
| Micallef, Kurt and |
| Claudia Borg", |
| booktitle = {Proceedings of the Fifteenth Biennial Language Resources and Evaluation Conference (LREC)}, |
| month = may, |
| year = "2026", |
| address = "Palma, Mallorca, Spain", |
| publisher = "European Language Resources Association", |
| } |
| ``` |
|
|
