Update README.md
Browse files
README.md
CHANGED
|
@@ -15,9 +15,8 @@ Lil-Bevo is UT Austin's submission to the BabyLM challenge, specifically the *st
|
|
| 15 |
## TLDR:
|
| 16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
| 17 |
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
|
| 18 |
-
- Model continues training for 50 epochs on 10M tokens with
|
| 19 |
-
- Model
|
| 20 |
-
- Model is trained with targeted linguistic masking for 10 epochs.
|
| 21 |
|
| 22 |
|
| 23 |
This README will be updated with more details soon.
|
|
|
|
| 15 |
## TLDR:
|
| 16 |
- Unigram tokenizer trained on 10M BabyLM tokens plus MAESTRO dataset for a vocab size of 16k.
|
| 17 |
- `deberta-small-v3` trained on mixture of MAESTRO and 10M tokens for 5 epochs.
|
| 18 |
+
- Model continues training for 50 epochs on 10M tokens with sequence length of 128.
|
| 19 |
+
- Model is trained for 2 epochs with targeted linguistic masking with sequence length of 512.
|
|
|
|
| 20 |
|
| 21 |
|
| 22 |
This README will be updated with more details soon.
|