| | --- |
| | language: en |
| | tags: |
| | - BabyBERTa |
| | license: mit |
| | datasets: |
| | - CHILDES |
| | widget: |
| | - text: "Look here. What is that <mask> ?" |
| | - text: "Do you like your <mask> ?" |
| | --- |
| | |
| | ## BabyBERTA |
| |
|
| | ### Overview |
| |
|
| | BabyBERTa is a light-weight version of RoBERTa trained on 5M words of American-English child-directed input. |
| | It is intended for language acquisition research, on a single desktop with a single GPU - no high-performance computing infrastructure needed. |
| |
|
| | The three provided models are randomly selected from 10 that were trained and reported in the paper. |
| |
|
| | ## Loading the tokenizer |
| |
|
| | BabyBERTa was trained with `add_prefix_space=True`, so it will not work properly with the tokenizer defaults. |
| | For instance, to load the tokenizer for BabyBERTa-1, load it as follows: |
| |
|
| | ```python |
| | tokenizer = RobertaTokenizerFast.from_pretrained("phueb/BabyBERTa-1", |
| | add_prefix_space=True) |
| | ``` |
| |
|
| | ### Hyper-Parameters |
| |
|
| | See the paper for details. |
| | All provided models were trained for 400K steps with a batch size of 16. |
| | Importantly, BabyBERTa never predicts unmasked tokens during training - `unmask_prob` is set to zero. |
| |
|
| |
|
| | ### Performance |
| |
|
| | BabyBerta was developed for learning grammatical knowledge from child-directed input. |
| | Its grammatical knowledge was evaluated using the [Zorro](https://github.com/phueb/Zorro) test suite. |
| | The best model achieves an overall accuracy of 80.3, |
| | comparable to RoBERTa-base, which achieves an overall accuracy of 82.6 on the latest version of Zorro (as of October, 2021). |
| | Both values differ slightly from those reported in the [CoNLL 2021 paper](https://aclanthology.org/2021.conll-1.49/). |
| | There are two reasons for this: |
| | 1. Performance of RoBERTa-base is slightly larger because the authors previously lower-cased all words in Zorro before evaluation. |
| | Lower-casing of proper nouns is detrimental to RoBERTa-base because RoBERTa-base has likely been trained on proper nouns that are primarily title-cased. |
| | In contrast, because BabyBERTa is not case-sensitive, its performance is not influenced by this change. |
| | 2. The latest version of Zorro no longer contains ambiguous content words such as "Spanish" which can be both a noun and an adjective. |
| | this resulted in a small reduction in the performance of BabyBERTa. |
| | |
| | Overall Accuracy on Zorro: |
| | |
| | | Model Name | Accuracy (holistic scoring) | Accuracy (MLM-scoring) | |
| | |----------------------------------------|------------------------------|------------| |
| | | [BabyBERTa-1][link-BabyBERTa-1] | 80.3 | 79.9 | |
| | | [BabyBERTa-2][link-BabyBERTa-2] | 78.6 | 78.2 | |
| | | [BabyBERTa-3][link-BabyBERTa-3] | 74.5 | 78.1 | |
| |
|
| |
|
| |
|
| | ### Additional Information |
| |
|
| | This model was trained by [Philip Huebner](https://philhuebner.com), currently at the [UIUC Language and Learning Lab](http://www.learninglanguagelab.org). |
| |
|
| | More info can be found [here](https://github.com/phueb/BabyBERTa). |
| |
|
| |
|
| | [link-BabyBERTa-1]: https://huggingface.co/phueb/BabyBERTa-1 |
| | [link-BabyBERTa-2]: https://huggingface.co/phueb/BabyBERTa-2 |
| | [link-BabyBERTa-3]: https://huggingface.co/phueb/BabyBERTa-3 |
| |
|
