misoRoBERTuito-2e

misoRoBERTuito-2e is a domain adaptation of RoBERTuito language model, specifically adapted to the misogyny domain.

It was adapted using a guided lexical masking strategy during masked language model (MLM) pretraining. Instead of randomly masking tokens, we prioritized masking words appearing in a misogyny-specific lexicon. The base corpus used for domain adaptation was the Spanish Hate Speech Superset.

For training the model we used a batch size of 8, with a learning rate of 2e-5. We trained the model for two epochs using a NVIDIA GeForce RTX 5090 GPU.

Usage

from transformers import pipeline
pipe = pipeline("fill-mask", model="citiusLTL/misoRoBERTuito-2e")
text = pipe("Ella es una<mask>")
print(text)

Load model directly

from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("citiusLTL/misoRoBERTuito-2e")
model = AutoModelForMaskedLM.from_pretrained("citiusLTL/misoRoBERTuito-2e")