| | --- |
| | license: cc-by-nc-4.0 |
| | language: |
| | - es |
| | base_model: |
| | - dccuchile/bert-base-spanish-wwm-uncased |
| | datasets: |
| | - manueltonneau/spanish-hate-speech-superset |
| | tags: |
| | - BETO |
| | - beto |
| | - hate_speech |
| | - immigrant |
| | - misogyny |
| | - BERT |
| | - spanish |
| | pipeline_tag: fill-mask |
| | library_name: transformers |
| | widget: |
| | - text: Los [MASK] son los causantes del aumento del desempleo |
| | --- |
| | # immisoBETO |
| |
|
| | immisoBETO is a domain adaptation of a [Spanish BERT](https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased) language model, specifically adapted to the immigrant and misogyny domain. |
| |
|
| | It was adapted using a guided lexical masking strategy during masked language model (MLM) pretraining. |
| | Instead of randomly masking tokens, we prioritized masking words appearing in a [immigrant](https://github.com/fmplaza/hate-speech-spanish-lexicons/blob/master/immigrant_lexicon.txt) and [misogyny](https://github.com/fmplaza/hate-speech-spanish-lexicons/blob/master/misogyny_lexicon.txt)-specific lexicon. |
| | The base corpus used for domain adaptation was the [Spanish Hate Speech Superset](https://huggingface.co/datasets/manueltonneau/spanish-hate-speech-superset). |
| |
|
| |
|
| | For training the model we used a batch size of 8, with a learning rate of 2e-5. We trained the model for four epochs using a NVIDIA GeForce RTX 5090 GPU. |
| |
|
| | ## Usage |
| |
|
| | ```python |
| | from transformers import pipeline |
| | pipe = pipeline("fill-mask", model="citiusLTL/immisoBETO") |
| | text = pipe("Los [MASK] son los causantes del aumento del desempleo") |
| | print(text) |
| | ``` |
| |
|
| | ## Load model directly |
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForMaskedLM |
| | tokenizer = AutoTokenizer.from_pretrained("citiusLTL/immisoBETO") |
| | model = AutoModelForMaskedLM.from_pretrained("citiusLTL/immisoBETO") |
| | ``` |
