Updating README

README.md

@@ -7,114 +7,44 @@ widget:
 ---
 # CAMeLBERT MSA NER Model
 ## Model description
-**CAMeLBERT MSA NER Model ** Named Entity Recognition (NER) model that was built by fine-tuning the [CAMeLBERT Modern Standard Arabic (MSA)](https://huggingface.co/CAMeL-Lab/bert-base-arabic-camelbert-msa/) model. For the fine-tuning, we used the [ANERcorp](https://camel.abudhabi.nyu.edu/anercorp/) dataset. Our fine-tuning procedure and the hyperparameters we used can be found in our paper *"[The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://arxiv.org/abs/2103.06678)."*
+**CAMeLBERT MSA NER Model** is Named Entity Recognition (NER) model that was built by fine-tuning the [CAMeLBERT Modern Standard Arabic (MSA)](https://huggingface.co/CAMeL-Lab/bert-base-arabic-camelbert-msa/) model. For the fine-tuning, we used the [ANERcorp](https://camel.abudhabi.nyu.edu/anercorp/) dataset. Our fine-tuning procedure and the hyperparameters we used can be found in our paper *"[The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://arxiv.org/abs/2103.06678)."* Our fine-tuning code can be found [here](https://github.com/CAMeL-Lab/CAMeLBERT).
 
 ## Intended uses
-You can use the released model for either masked language modeling or next sentence prediction.
-However, it is mostly intended to be fine-tuned on an NLP task, such as NER, POS tagging, sentiment analysis, dialect identification, and poetry classification.
-We release our fine-tuninig code [here](https://github.com/CAMeL-Lab/CAMeLBERT).
+You can use the CAMeLBERT MSA NER Model directly as part of the transformers pipeline.
+
 #### How to use
-You can use this model directly with a pipeline for masked language modeling:
+You can use this model directly with a pipeline to do NER:
 ```python
 >>> from transformers import pipeline
->>> unmasker = pipeline('fill-mask', model='CAMeL-Lab/bert-base-arabic-camelbert-msa')
->>> unmasker("الهدف من الحياة هو [MASK] .")
-[{'sequence': '[CLS] الهدف من الحياة هو العمل. [SEP]',
-  'score': 0.08507660031318665,
-  'token': 2854,
-  'token_str': 'العمل'},
- {'sequence': '[CLS] الهدف من الحياة هو الحياة. [SEP]',
-  'score': 0.058905381709337234,
-  'token': 3696, 'token_str': 'الحياة'},
- {'sequence': '[CLS] الهدف من الحياة هو النجاح. [SEP]',
-  'score': 0.04660581797361374, 'token': 6232,
-  'token_str': 'النجاح'},
- {'sequence': '[CLS] الهدف من الحياة هو الربح. [SEP]',
-  'score': 0.04156001657247543,
-  'token': 12413, 'token_str': 'الربح'},
- {'sequence': '[CLS] الهدف من الحياة هو الحب. [SEP]',
-  'score': 0.03534102067351341,
-  'token': 3088,
-  'token_str': 'الحب'}]
-```
-*Note*: to download our models, you would need `transformers>=3.5.0`. Otherwise, you could download the models manually.
-Here is how to use this model to get the features of a given text in PyTorch:
-```python
-from transformers import AutoTokenizer, AutoModel
-tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa')
-model = AutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa')
-text = "مرحبا يا عالم."
-encoded_input = tokenizer(text, return_tensors='pt')
-output = model(**encoded_input)
+>>> ner = pipeline('ner', model='CAMeL-Lab/bert-base-arabic-camelbert-msa-ner')
+>>> ner("إمارة أبوظبي هي إحدى إمارات دولة الإمارات العربية المتحدة السبع")
+[{'word': 'أبوظبي',
+  'score': 0.9895730018615723,
+  'entity': 'B-LOC',
+  'index': 2,
+  'start': 6,
+  'end': 12},
+ {'word': 'الإمارات',
+  'score': 0.8156259655952454,
+  'entity': 'B-LOC',
+  'index': 8,
+  'start': 33,
+  'end': 41},
+ {'word': 'العربية',
+  'score': 0.890906810760498,
+  'entity': 'I-LOC',
+  'index': 9,
+  'start': 42,
+  'end': 49},
+ {'word': 'المتحدة',
+  'score': 0.8169114589691162,
+  'entity': 'I-LOC',
+  'index': 10,
+  'start': 50,
+  'end': 57}]
 ```
-and in TensorFlow:
-```python
-from transformers import AutoTokenizer, TFAutoModel
-tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa')
-model = TFAutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa')
-text = "مرحبا يا عالم."
-encoded_input = tokenizer(text, return_tensors='tf')
-output = model(encoded_input)
-```
-## Training data
-- MSA (Modern Standard Arabic)
-  - [The Arabic Gigaword Fifth Edition](https://catalog.ldc.upenn.edu/LDC2011T11)
-  - [Abu El-Khair Corpus](http://www.abuelkhair.net/index.php/en/arabic/abu-el-khair-corpus)
-  - [OSIAN corpus](https://vlo.clarin.eu/search;jsessionid=31066390B2C9E8C6304845BA79869AC1?1&q=osian)
-  - [Arabic Wikipedia](https://archive.org/details/arwiki-20190201)
-  - The unshuffled version of the Arabic [OSCAR corpus](https://oscar-corpus.com/)
-## Training procedure
-We use [the original implementation](https://github.com/google-research/bert) released by Google for pre-training.
-We follow the original English BERT model's hyperparameters for pre-training, unless otherwise specified.
-### Preprocessing
-- After extracting the raw text from each corpus, we apply the following pre-processing.
-- We first remove invalid characters and normalize white spaces using the utilities provided by [the original BERT implementation](https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/tokenization.py#L286-L297).
-- We also remove lines without any Arabic characters.
-- We then remove diacritics and kashida using [CAMeL Tools](https://github.com/CAMeL-Lab/camel_tools).
-- Finally, we split each line into sentences with a heuristics-based sentence segmenter.
-- We train a WordPiece tokenizer on the entire dataset (167 GB text) with a vocabulary size of 30,000 using [HuggingFace's tokenizers](https://github.com/huggingface/tokenizers).
-- We do not lowercase letters nor strip accents.
-### Pre-training
-- The model was trained on a single cloud TPU (`v3-8`) for one million steps in total.
-- The first 90,000 steps were trained with a batch size of 1,024 and the rest was trained with a batch size of 256.
-- The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%.
-- We use whole word masking and a duplicate factor of 10.
-- We set max predictions per sequence to 20 for the dataset with max sequence length of 128 tokens and 80 for the dataset with max sequence length of 512 tokens.
-- We use a random seed of 12345, masked language model probability of 0.15, and short sequence probability of 0.1.
-- The optimizer used is Adam with a learning rate of 1e-4, \\(\beta_{1} = 0.9\\) and \\(\beta_{2} = 0.999\\), a weight decay of 0.01, learning rate warmup for 10,000 steps and linear decay of the learning rate after.
-## Evaluation results
-- We evaluate our pre-trained language models on five NLP tasks: NER, POS tagging, sentiment analysis, dialect identification, and poetry classification.
-- We fine-tune and evaluate the models using 12 dataset.
-- We used Hugging Face's transformers to fine-tune our CAMeLBERT models.
-- We used transformers `v3.1.0` along with PyTorch `v1.5.1`.
-- The fine-tuning was done by adding a fully connected linear layer to the last hidden state.
-- We use \\(F_{1}\\) score as a metric for all tasks.
-- Code used for fine-tuning is available [here](https://github.com/CAMeL-Lab/CAMeLBERT).
-### Results
-| Task                 | Dataset         | Variant | Mix   | CA    | DA    | MSA   | MSA-1/2 | MSA-1/4 | MSA-1/8 | MSA-1/16 |
-| -------------------- | --------------- | ------- | ----- | ----- | ----- | ----- | ------- | ------- | ------- | -------- |
-| NER                  | ANERcorp        | MSA     | 80.8% | 67.9% | 74.1% | 82.4% | 82.0%   | 82.1%   | 82.6%   | 80.8%    |
-| POS                  | PATB (MSA)      | MSA     | 98.1% | 97.8% | 97.7% | 98.3% | 98.2%   | 98.3%   | 98.2%   | 98.2%    |
-|                      | ARZTB (EGY)     | DA      | 93.6% | 92.3% | 92.7% | 93.6% | 93.6%   | 93.7%   | 93.6%   | 93.6%    |
-|                      | Gumar (GLF)     | DA      | 97.3% | 97.7% | 97.9% | 97.9% | 97.9%   | 97.9%   | 97.9%   | 97.9%    |
-| SA                   | ASTD            | MSA     | 76.3% | 69.4% | 74.6% | 76.9% | 76.0%   | 76.8%   | 76.7%   | 75.3%    |
-|                      | ArSAS           | MSA     | 92.7% | 89.4% | 91.8% | 93.0% | 92.6%   | 92.5%   | 92.5%   | 92.3%    |
-|                      | SemEval         | MSA     | 69.0% | 58.5% | 68.4% | 72.1% | 70.7%   | 72.8%   | 71.6%   | 71.2%    |
-| DID                  | MADAR-26        | DA      | 62.9% | 61.9% | 61.8% | 62.6% | 62.0%   | 62.8%   | 62.0%   | 62.2%    |
-|                      | MADAR-6         | DA      | 92.5% | 91.5% | 92.2% | 91.9% | 91.8%   | 92.2%   | 92.1%   | 92.0%    |
-|                      | MADAR-Twitter-5 | MSA     | 75.7% | 71.4% | 74.2% | 77.6% | 78.5%   | 77.3%   | 77.7%   | 76.2%    |
-|                      | NADI            | DA      | 24.7% | 17.3% | 20.1% | 24.9% | 24.6%   | 24.6%   | 24.9%   | 23.8%    |
-| Poetry               | APCD            | CA      | 79.8% | 80.9% | 79.6% | 79.7% | 79.9%   | 80.0%   | 79.7%   | 79.8%    |
-### Results (Average)
-|                      | Variant | Mix   | CA    | DA    | MSA   | MSA-1/2 | MSA-1/4 | MSA-1/8 | MSA-1/16 |
-| -------------------- | ------- | ----- | ----- | ----- | ----- | ------- | ------- | ------- | -------- |
-| Variant-wise-average<sup>[[1]](#footnote-1)</sup> | MSA     | 82.1% | 75.7% | 80.1% | 83.4% | 83.0%   | 83.3%   | 83.2%   | 82.3%    |
-|                      | DA      | 74.4% | 72.1% | 72.9% | 74.2% | 74.0%   | 74.3%   | 74.1%   | 73.9%    |
-|                      | CA      | 79.8% | 80.9% | 79.6% | 79.7% | 79.9%   | 80.0%   | 79.7%   | 79.8%    |
-| Macro-Average        | ALL     | 78.7% | 74.7% | 77.1% | 79.2% | 79.0%   | 79.2%   | 79.1%   | 78.6%    |
-<a name="footnote-1">[1]</a>: Variant-wise-average refers to average over a group of tasks in the same language variant.
-## Acknowledgements
-This research was supported with Cloud TPUs from Google’s TensorFlow Research Cloud (TFRC).
+*Note*: to download our models, you would need `transformers>=3.5.0`. Otherwise, you could download the models 
+
 ## Citation
 ```bibtex
 @inproceedings{inoue-etal-2021-interplay,