Create README.md

README.md

@@ -0,0 +1,64 @@
+---
+language: es
+tags:
+- Spanish
+- BART
+- biology
+- medical
+- seq2seq
+license: mit
+---
+
+##  BIOMEDtra 
+
+**BIOMEDtra** (small) is an Electra like model (discriminator in this case) trained on 
+
+
+## 🦠 NarbioBART 🏥 
+
+**NarbioBART** (base) is a BART-like model trained on [Spanish Biomedical Crawled Corpus](https://zenodo.org/record/5510033#.Yhdk1ZHMLJx).
+
+BART is a transformer *encoder-decoder* (seq2seq) model with a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder. BART is pre-trained by (1) corrupting text with an arbitrary noising function and (2) learning a model to reconstruct the original text.
+
+This model is particularly effective when fine-tuned for text generation tasks (e.g., summarization, translation) but also works well for comprehension tasks (e.g., text classification, question answering).
+
+
+
+## Training details
+
+- Dataset: `Spanish Biomedical Crawled Corpus` - 90% for training / 10% for validation.
+- Training script: see [here](https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_bart_dlm_flax.py)
+
+
+## [Evaluation metrics](https://huggingface.co/mrm8488/bart-bio-base-es/tensorboard?params=scalars#frame) 🧾
+
+|Metric | # Value |
+|-------|---------|
+|Accuracy| 0.802|
+|Loss| 1.04|
+
+
+## Benchmarks 🔨
+
+WIP 🚧
+
+## How to use with `transformers`
+
+```py
+from transformers import BartForConditionalGeneration, BartTokenizer
+
+model_id = "Narrativa/NarbioBART"
+
+model = BartForConditionalGeneration.from_pretrained(model_id, forced_bos_token_id=0)
+tokenizer = BartTokenizer.from_pretrained(model_id)
+
+def fill_mask_span(text):
+  batch = tokenizer(text, return_tensors="pt")
+  generated_ids = model.generate(batch["input_ids"])
+  print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True))
+
+text = "your text with a <mask> token."
+fill_mask_span(text)
+```
+
+## Citation