Update README.md

README.md

@@ -11,4 +11,109 @@ tags:
 
 # Sci-BETO-base
 
-**Sci-BETO** (*Scientific BETO*) es un modelo tipo RoBERTa entrenado desde cero sobre un corpus de textos científicos en español.
+**Sci-BETO** is a domain-specific RoBERTa encoder pretrained entirely on **Spanish scientific texts**.  
+
+---
+
+## Model Description
+
+Sci-BETO-base is a transformer-based encoder following the **RoBERTa** architecture (125M parameters).  
+It was pretrained from scratch using byte-level BPE tokenization on a large corpus of **Spanish open-access scientific publications**, including theses, dissertations, and peer-reviewed papers from Colombian universities and international repositories.
+
+The model was designed to capture **scientific discourse**, terminology, and abstract reasoning patterns typical of research documents in economics, engineering, medicine, and the social sciences.
+
+| Property | Value |
+|-----------|--------|
+| Architecture | RoBERTa-base |
+| Parameters | ~125M |
+| Vocabulary size | 50,262 |
+| Tokenizer | Byte-Level BPE (trained from scratch) |
+| Pretraining objective | Masked Language Modeling (MLM) |
+| Pretraining steps | 85K |
+| Max sequence length | 512 tokens |
+| Framework | Transformers |
+
+---
+
+## Pretraining Data
+
+The pretraining corpus includes over **11 billion tokens** from Spanish academic and scientific sources:
+
+- Open-access repositories of Colombian universities (Universidad de los Andes, Universidad Nacional, Universidad Javeriana, Universidad del Rosario).  
+- CORE API and institutional repositories (theses, dissertations, working papers).
+- Tax Statutes in Colombia
+- Filtered and cleaned with:
+  - Language detection using FastText,
+  - Deduplication,
+  - Abstract and reference removal,
+  - Normalization of encoding and special symbols.
+
+The final dataset covers multiple disciplines (economics, medicine, engineering, humanities), ensuring representation across scientific domains.
+
+| **Source**                     | **# Documents** | **# Words (deduplicated)** | **Percentage (%)** |
+|--------------------------------|----------------:|----------------------------:|-------------------:|
+| Universidad de los Andes        | 33,858          | 365,752,780                 | 3.23               |
+| Universidad Nacional            | 44,686          | 537,022,975                 | 4.75               |
+| CORE API                        | 2,181,689       | 9,624,189,002               | 85.10              |
+| Universidad del Rosario         | 22,404          | 183,356,109                 | 1.62               |
+| Universidad Javeriana           | 25,624          | 323,918,445                 | 2.86               |
+| Tax Statutes in Colombia        | 392             | 13,924,060                  | 0.12               |
+| Extra                           | 2               | 261,131,453                 | 2.31               |
+| **Total**                       | **2,308,655**   | **11,309,295,824**          | **100.00**         |
+
+---
+
+## Benchmarks
+
+Sci-BETO was fine-tuned and benchmarked across multiple downstream tasks, both general-domain and scientific:
+
+| **Dataset** | **Metric** | **Sci-BETO Large** | **Sci-BETO Base** | **BETO** | **BERTIN** | **XLM-RoBERTa Large** |
+|--------------|-------------|-------------------:|------------------:|----------:|------------:|----------------------:|
+| **WikiCAT**       | F1 (macro)       | **0.7738** | 0.7583 | 0.7624 | 0.7598 | — |
+| **PAWS-X (es)**   | F1 (macro)       | **0.9148** | 0.8794 | 0.8985 | 0.8961 | — |
+| **PharmaCoNER**   | F1 (micro)       | **0.8959** | 0.8733 | 0.8845 | 0.8802 | — |
+| **CANTEMIST**     | F1 (micro)       | 0.8809 | 0.8784 | 0.8954 | **0.8956** | — |
+| **NLI (ESNLI-R)** | F1 (micro)       | — | — | — | — | — |
+| **BanRep (JEL)**  | Exact Match      | **0.6116** | 0.6043 | 0.5933 | 0.5807 | — |
+| **Rosario**       | F1 (macro)       | **0.9203** | 0.9194 | 0.9079 | 0.9121 | — |
+| **Econ-IE**       | F1 (micro)       | **0.5256** | 0.5158 | 0.5199 | 0.4992 | — |
+
+On average, **Sci-BETO** achieves comparable or superior results to general-domain Spanish models in specialized contexts (scientific, biomedical, economic), while maintaining strong performance in general text understanding.
+
+---
+
+## Intended Use
+
+- Research and experimentation in **Spanish scientific NLP**.
+- Downstream fine-tuning for:
+  - Text classification (scientific or academic domains),
+  - Named Entity Recognition (NER),
+  - Semantic similarity and paraphrase detection,
+  - Knowledge extraction from academic documents.
+
+---
+
+## Limitations
+
+- The model may underperform on highly informal or non-academic Spanish (e.g., social media).  
+- It is not designed for generative tasks (e.g., text completion, chat).  
+- Domain bias toward academic register and Latin American Spanish variants.  
+- Pretraining corpus excludes English or bilingual data.
+
+---
+
+## Example Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+
+tokenizer = AutoTokenizer.from_pretrained("Flaglab/Sci-BETO-base")
+model = AutoModelForMaskedLM.from_pretrained("Flaglab/Sci-BETO-base")
+
+text = "El Banco de la Republica va a subir las [mask] de interes."
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+logits = outputs.logits
+masked_index = (inputs.input_ids == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
+predicted_token = tokenizer.decode(logits[0, masked_index].argmax(dim=-1))
+print("Predicted token:", predicted_token)