Update model card

README.md

@@ -7,20 +7,68 @@ tags:
 - OpenCENIA
 datasets:
 - large_spanish_corpus
+pipeline_tag: fill-mask
+widget:
+- text: Quito es la capital de [MASK].
 ---
 
-# ALBERT Tiny Spanish
+# ALBETO (tiny-sized model)
 
-This is an [ALBERT](https://github.com/google-research/albert) model trained on a [big spanish corpora](https://github.com/josecannete/spanish-corpora).
-The model was trained on a single TPU v3-8 with the following hyperparameters and steps/time:
-- LR: 0.00125
-- Batch Size: 2048
-- Warmup ratio: 0.0125
-- Warmup steps: 125000
-- Goal steps: 10000000
-- Total steps: 8300000
-- Total training time (aprox): 58.2 days
+ALBETO model pre-trained on Spanish language. It was introduced in the paper [ALBETO and DistilBETO: Lightweight Spanish Language Models](https://arxiv.org/abs/2204.09145).
 
-## Training loss
+## Model description
 
-![https://drive.google.com/uc?export=view&id=1KQc8yWZLKvDLjBtu4IOAgpTx0iLcvX_Q](https://drive.google.com/uc?export=view&id=1KQc8yWZLKvDLjBtu4IOAgpTx0iLcvX_Q)
+ALBETO is an [ALBERT](https://github.com/google-research/albert) model (transformer encoder) pre-trained exclusively on a [Spanish corpus](https://github.com/josecannete/spanish-corpora) of about 3B words, the same data used for BETO. ALBERT factorizes the embedding layer and shares parameters across layers, so ALBETO needs far fewer parameters than BERT-style models of similar depth.
+
+| Parameters | Layers | Hidden size | Embedding size | Vocabulary |
+|------------|--------|-------------|----------------|------------|
+| 5M | 4 | 312 | 128 | 31K (uncased) |
+
+It was trained with LAMB for 8.3M steps (batch size 2,048, learning rate 1.25e-3) on a single TPU v3-8 (~58 days).
+
+## Intended uses & limitations
+
+You can use the raw model for masked language modeling. However, the model is mainly meant to be fine-tuned on a supervised dataset (e.g. text classification, token classification, question answering).
+
+The model is uncased.
+
+### How to use
+
+Here is how to use this model in PyTorch:
+
+```python
+from transformers import PreTrainedTokenizerFast, AutoModel
+
+tokenizer = PreTrainedTokenizerFast.from_pretrained('dccuchile/albert-tiny-spanish')
+model = AutoModel.from_pretrained('dccuchile/albert-tiny-spanish')
+
+inputs = tokenizer("Hola amigo, bienvenido a casa.", return_tensors="pt")
+outputs = model(**inputs)
+
+last_hidden_states = outputs.last_hidden_state
+```
+
+Load the tokenizer with `PreTrainedTokenizerFast`: `AutoTokenizer` fails on this model with transformers 5.x.
+
+## Evaluation results
+
+| POS | NER | MLDoc | PAWS-X | XNLI | MLQA | SQAC | TAR, XQuAD |
+|-----|-----|-------|--------|------|------|------|------------|
+| 97.04 | 75.11 | 95.82 | 80.20 | 73.43 | 51.84 / 28.28 | 59.28 / 39.16 | 66.43 / 45.71 |
+
+POS/NER: F1. MLDoc/PAWS-X/XNLI: accuracy. QA: F1 / exact match. The [GitHub repository](https://github.com/dccuchile/lightweight-spanish-language-models) compares all models with BETO and links the fine-tuned checkpoints.
+
+### Citation (BibTeX)
+
+```bibtex
+@inproceedings{canete-etal-2022-albeto,
+    title = "{ALBETO} and {D}istil{BETO}: Lightweight {S}panish Language Models",
+    author = "Ca{\~n}ete, Jos{\'e} and Donoso, Sebasti{\'a}n and Bravo-Marquez, Felipe and Carvallo, Andr{\'e}s and Araujo, Vladimir",
+    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
+    year = "2022",
+    address = "Marseille, France",
+    publisher = "European Language Resources Association",
+    url = "https://aclanthology.org/2022.lrec-1.457/",
+    pages = "4291--4298"
+}
+```