nlpie
/

tiny-biobert

Model card Files Files and versions

omidrohanian commited on Nov 4, 2022

Commit

e5b80b3

·

1 Parent(s): 1638c07

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -5,7 +5,7 @@ TinyBioBERT is a distilled version of the [BioBERT](https://huggingface.co/dmis-
 This model uses a unique distillation method called ‘transformer-layer distillation’ which is applied on each layer of the student to align the attention maps and the hidden states of the student with those of the teacher.
 # Architecture and Initialisation
-This model uses 4 hidden layers with a hidden dimension size and an embedding size of 768 resulting in a total of 15M parameters. Due to the small hidden dimension size used in this model, it uses a random initialisation.
 # Citation

 This model uses a unique distillation method called ‘transformer-layer distillation’ which is applied on each layer of the student to align the attention maps and the hidden states of the student with those of the teacher.
 # Architecture and Initialisation
+This model uses 4 hidden layers with a hidden dimension size and an embedding size of 768 resulting in a total of 15M parameters. Due to the model's small hidden dimension size, it uses random initialisation.
 # Citation