Upload README.md

README.md

@@ -20,6 +20,90 @@ Bioformer-8L was pre-trained from scratch on the same corpus as the vocabulary (
 
 Pre-training of Bioformer-8L was performed on a single Cloud TPU device (TPUv2, 8 cores, 8GB memory per core). The maximum input sequence length was fixed to 512, and the batch size was set to 256. We pre-trained Bioformer-8L for 2 million steps, which took about 8.3 days.
 
+## Usage
+
+Prerequisites: python3, pytorch, transformers and datasets
+
+We have tested the following commands on Python v3.9.16, PyTorch v1.13.1+cu117, Datasets v2.9.0 and Transformers v4.26.
+
+To install pytorch, please refer to instructions [here](https://pytorch.org/get-started/locally).
+
+To install the `transformers` and `datasets` library:
+```
+pip install transformers
+pip install datasets
+```
+
+### Filling mask
+
+```
+from transformers import pipeline
+unmasker8L = pipeline('fill-mask', model='bioformers/bioformer-8L')
+unmasker8L("[MASK] refers to a group of diseases that affect how the body uses blood sugar (glucose)")
+
+unmasker16L = pipeline('fill-mask', model='bioformers/bioformer-16L')
+unmasker16L("[MASK] refers to a group of diseases that affect how the body uses blood sugar (glucose)")
+
+```
+
+Output of `bioformer-8L`:
+
+```
+[{'score': 0.3207533359527588, 
+'token': 13473, 
+'token_str': 'Diabetes', 
+'sequence': 'Diabetes refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}, 
+
+{'score': 0.19234347343444824, 
+'token': 17740, 
+'token_str': 'Obesity', 
+'sequence': 'Obesity refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}, 
+
+{'score': 0.09200277179479599, 
+'token': 10778, 
+'token_str': 'T2DM', 
+'sequence': 'T2DM refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}, 
+
+{'score': 0.08494312316179276, 
+'token': 2228, 
+'token_str': 'It', 
+'sequence': 'It refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}, 
+
+{'score': 0.0412776917219162, 
+'token': 22263, 
+'token_str': 
+'Hypertension', 
+'sequence': 'Hypertension refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}]
+```
+
+Output of `bioformer-16L`:
+
+```
+[{'score': 0.7262957692146301,
+'token': 13473,
+'token_str': 'Diabetes',
+'sequence': 'Diabetes refers to a group of diseases that affect how the body uses blood sugar ( glucose )'},
+
+{'score': 0.124954953789711,
+'token': 10778,
+'token_str': 'T2DM',
+'sequence': 'T2DM refers to a group of diseases that affect how the body uses blood sugar ( glucose )'},
+
+{'score': 0.04062706232070923,
+'token': 2228,
+'token_str': 'It',
+'sequence': 'It refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}, 
+
+{'score': 0.022694870829582214,
+'token': 17740,
+'token_str': 'Obesity',
+'sequence': 'Obesity refers to a group of diseases that affect how the body uses blood sugar ( glucose )'},
+
+{'score': 0.009743048809468746,
+'token': 13960,
+'token_str': 'T2D',
+'sequence': 'T2D refers to a group of diseases that affect how the body uses blood sugar ( glucose )'}]
+```
 
 ## Awards
 Bioformer-8L achieved top performance (highest micro-F1 score) in the BioCreative VII COVID-19 multi-label topic classification challenge (https://doi.org/10.1093/database/baac069)