File size: 1,249 Bytes
e9b4790 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 | ---
license: apache-2.0
language:
- en
base_model:
- bsu-slim/electra-tiny
- lgcharpe/ELC_BERT_small_baby_10M
pipeline_tag: text-classification
library_name: transformers
---
# This model is currently experimental and broken!
A pretrained [ELECTRA-Tiny](https://huggingface.co/bsu-slim/electra-tiny/tree/main) model modified to implement zero initialization
transformer layer weighting as described in
[Not all layers are equally as important: Every Layer Counts BERT](https://aclanthology.org/2023.conll-babylm.20.pdf).
# Training
Used pretraining pipeline as defined in this [repository](https://github.com/bakirgrbic/bblm).
## Hyperparameters
- Epochs: 9
- Batch size: 8
- Learning rate: 1e-4
- Optimizer: AdamW
## Resources Used
- Compute: AWS Sagemaker ml.g4dn.xlarge
- Time: About 63 hours
# Evaluation
## BLiMP
Used BLiMP evaluation from the [2024 BabyLM evaluation pipeline repository](https://github.com/babylm/evaluation-pipeline-2024).
### Results
- blimp_supplement accuracy: 47.54%
- blimp_filtered accuracy: 51.79%
- See [blimp_results](./blimp_results) for a detailed breakdown on subtasks.
### Hyperparameters
- Epochs: 1
- Script modified for masked LMs
### Resources Used
- Compute: arm64 MacOS
- Time: About 30 minutes |