Create README.md

README.md

@@ -0,0 +1,56 @@
+---
+language: 
+  - es
+thumbnail: "url to a thumbnail used in social sharing"
+tags:
+- tag1
+- tag2
+license: apache-2.0
+datasets:
+- Oscar
+metrics:
+- metric1
+- metric2
+---
+
+# SELECTRA: A Spanish ELECTRA
+
+SELECTRA is a Spanish pre-trained language model based on [ELECTRA](https://github.com/google-research/electra).
+We release a `small` and `medium` version with the following configuration:
+
+| Model | Layers | Embedding/Hidden Size | Params | Vocab Size | Max Sequence Length | Cased |
+| --- | --- | --- | --- | ---  | --- | --- |
+| SELECTRA small | 12 | 256 | 22M | 50k | 512 | True |
+| SELECTRA medium | 12 | 384 | 41M | 50k | 512 | True |
+
+## Usage
+
+```python
+from transformers import ElectraForPreTraining, ElectraTokenizerFast
+
+discriminator = ElectraForPreTraining.from_pretrained("models/small/pytorch_model")
+tokenizer = ElectraTokenizerFast.from_pretrained("models/medium/pytorch_model")
+```
+- Links to our zero-shot-classifiers
+
+## Metrics
+
+We fine-tune our models on 4 different down-stream tasks:
+
+ - [XNLI](https://huggingface.co/datasets/xnli)
+ - [PAWS-X](https://huggingface.co/datasets/paws-x)
+ - [CoNLL2002 - POS](https://huggingface.co/datasets/conll2002)
+ - [CoNLL2002 - NER](https://huggingface.co/datasets/conll2002)
+ 
+We provide the mean and standard deviation of 5 fine-tuning runs. 
+| Model | 
+|
+ 
+
+## Training  
+
+- Link to our repo
+
+## Motivation
+
+Despite the abundance of excelent Spanish language models (BETO, bertin, etc) we felt there was still a lack of distilled or compact models with comparable metrics to their bigger siblings.