---
language: es
library_name: transformers
license: apache-2.0
tags:
  - roberta
  - spanish
  - scientific
  - fill-mask
---

# Sci-BETO-base

**Sci-BETO** is a domain-specific RoBERTa encoder pretrained entirely on **Spanish scientific texts**.  

---

## Model Description

Sci-BETO-base is a transformer-based encoder following the **RoBERTa** architecture (125M parameters).  
It was pretrained from scratch using byte-level BPE tokenization on a large corpus of **Spanish open-access scientific publications**, including theses, dissertations, and peer-reviewed papers from Colombian universities and international repositories.

The model was designed to capture **scientific discourse**, terminology, and abstract reasoning patterns typical of research documents in economics, engineering, medicine, and the social sciences.

| Property | Value |
|-----------|--------|
| Architecture | RoBERTa-base |
| Parameters | ~125M |
| Vocabulary size | 50,262 |
| Tokenizer | Byte-Level BPE (trained from scratch) |
| Pretraining objective | Masked Language Modeling (MLM) |
| Pretraining steps | 85K |
| Max sequence length | 512 tokens |
| Framework | Transformers |

---

## Pretraining Data

The pretraining corpus includes over **11 billion tokens** from Spanish academic and scientific sources:

- Open-access repositories of Colombian universities (Universidad de los Andes, Universidad Nacional, Universidad Javeriana, Universidad del Rosario).  
- CORE API and institutional repositories (theses, dissertations, working papers).
- Tax Statutes in Colombia

The final dataset covers multiple disciplines (economics, medicine, engineering, humanities), ensuring representation across scientific domains.

| **Source**                     | **# Documents** | **# Words (deduplicated)** | **Percentage (%)** |
|--------------------------------|----------------:|----------------------------:|-------------------:|
| Universidad de los Andes        | 33,858          | 365,752,780                 | 3.23               |
| Universidad Nacional            | 44,686          | 537,022,975                 | 4.75               |
| CORE API                        | 2,181,689       | 9,624,189,002               | 85.10              |
| Universidad del Rosario         | 22,404          | 183,356,109                 | 1.62               |
| Universidad Javeriana           | 25,624          | 323,918,445                 | 2.86               |
| Tax Statutes in Colombia        | 392             | 13,924,060                  | 0.12               |
| Extra                           | 2               | 261,131,453                 | 2.31               |
| **Total**                       | **2,308,655**   | **11,309,295,824**          | **100.00**         |

---

## Benchmarks

Sci-BETO was fine-tuned and benchmarked across multiple downstream tasks, both general-domain and scientific:

| **Dataset**        | **Metric**      | **Sci-BETO Large** | **Sci-BETO Base** | **BETO** | **BERTIN** |
|---------------------|----------------|-------------------:|------------------:|----------:|------------:|
| **WikiCAT**         | F1 (macro)     | **0.7738**         | 0.7583            | 0.7624    | 0.7598      |
| **PAWS-X (es)**     | F1 (macro)     | **0.9148**         | 0.8794            | 0.8985    | 0.8961      |
| **PharmaCoNER**     | F1 (micro)     | **0.8959**         | 0.8733            | 0.8845    | 0.8802      |
| **CANTEMIST**       | F1 (micro)     | 0.8809             | 0.8784            | 0.8954    | **0.8956**  |
| **NLI (ESNLI-R)**   | F1 (micro)     | —                  | —                 | —         | —           |
| **BanRep (JEL)**    | Exact Match    | **0.6116**         | 0.6043            | 0.5933    | 0.5807      |
| **Rosario**         | F1 (macro)     | **0.9203**         | 0.9194            | 0.9079    | 0.9121      |
| **Econ-IE**         | F1 (micro)     | **0.5256**         | 0.5158            | 0.5199    | 0.4992      |


On average, **Sci-BETO** achieves comparable or superior results to general-domain Spanish models in specialized contexts (scientific, biomedical, economic), while maintaining strong performance in general text understanding.

---

## Intended Use

- Research and experimentation in **Spanish scientific NLP**.
- Downstream fine-tuning for:
  - Text classification (scientific or academic domains),
  - Named Entity Recognition (NER),
  - Semantic similarity and paraphrase detection,
  - Knowledge extraction from academic documents.

---

## Limitations

- The model may underperform on highly informal or non-academic Spanish (e.g., social media).  
- It is not designed for generative tasks (e.g., text completion, chat).  
- Domain bias toward academic register and Latin American Spanish variants.  
- Pretraining corpus excludes English or bilingual data.

---

## Example Usage

```python
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("Flaglab/Sci-BETO-base")
model = AutoModelForMaskedLM.from_pretrained("Flaglab/Sci-BETO-base")

text = "El Banco de la República va a subir las [mask] de interes."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
masked_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_token = tokenizer.decode(logits[0, masked_index].argmax(dim=-1))
print("Predicted token:", predicted_token)