Spaces:
Running
Running
readme: emojify more headers π
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ The following LMs are currently supported:
|
|
| 17 |
* [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
|
| 18 |
* [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
|
| 19 |
|
| 20 |
-
# FineWeb-LMs
|
| 21 |
|
| 22 |
Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
|
| 23 |
|
|
@@ -25,7 +25,7 @@ Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https
|
|
| 25 |
* Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
|
| 26 |
* TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
|
| 27 |
|
| 28 |
-
# ScandEval Evaluation
|
| 29 |
|
| 30 |
To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
|
| 31 |
|
|
|
|
| 17 |
* [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
|
| 18 |
* [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
|
| 19 |
|
| 20 |
+
# π· FineWeb-LMs
|
| 21 |
|
| 22 |
Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
|
| 23 |
|
|
|
|
| 25 |
* Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
|
| 26 |
* TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
|
| 27 |
|
| 28 |
+
# π ScandEval Evaluation
|
| 29 |
|
| 30 |
To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
|
| 31 |
|