Create README.md

README.md

@@ -0,0 +1,21 @@
+---
+language: 
+- et
+
+license: cc-by-sa-4.0
+---
+# Usage
+Load in transformers library with:
+```
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+  
+  tokenizer = AutoTokenizer.from_pretrained("EMBEDDIA/est-roberta", use_fast=False)
+  model = AutoModelForMaskedLM.from_pretrained("EMBEDDIA/est-roberta")
+```
+**NOTE**: it is currently *critically important* to add `use_fast=False` parameter to tokenizer if using transformers version 4+ (prior versions have `use_fast=False` as default) By default it attempts to load a fast tokenizer, which might work (ie. not result in an error), but not correctly, as there is no current support for fast tokenizers for Camembert-based models.
+
+# Est-RoBERTa
+Est-RoBERTa model is a monolingual Estonian BERT-like model. It is closely related to French Camembert model https://camembert-model.fr/. The Estonian corpora used for training the model have 2.51 billion tokens in total. The subword vocabulary contains 40,000 tokens.
+
+Est-RoBERTa was trained for 40 epochs.
+