BERToli 🎶🇮🇹

About the model

BERToli is a BERT model for Italian song lyrics. It was obtained via continued pretraining of dbmdz/bert-base-italian-xxl-cased on ~106k Italian song lyrics from the Genius Song Lyrics Dataset. The objective was Masked Language Modeling (MLM).

The training code is available on GitHub.

Evaluation

The base model and the adapted model were tested on a held-out set of ~6k songs with the following results:

Model	MLM Loss	Perplexity
Base	1.94	6.95
BERToli	1.45	4.26

Evaluation of the learned representations will be made available in the future, once a suitable dataset has been created / identified.

Why BERToli?

Pierangelo Bertoli (5 November 1942 – 7 October 2002) was an Italian singer-songwriter and poet.