---
license: apache-2.0
language:
- en
base_model:
- bsu-slim/electra-tiny
- lgcharpe/ELC_BERT_small_baby_10M
pipeline_tag: text-classification
library_name: transformers
---
# This model is currently experimental and broken!

A pretrained [ELECTRA-Tiny](https://huggingface.co/bsu-slim/electra-tiny/tree/main) model modified to implement zero initialization
transformer layer weighting as described in 
[Not all layers are equally as important: Every Layer Counts BERT](https://aclanthology.org/2023.conll-babylm.20.pdf).


# Training
Used pretraining pipeline as defined in this [repository](https://github.com/bakirgrbic/bblm).

## Hyperparameters
- Epochs: 9
- Batch size: 8
- Learning rate: 1e-4
- Optimizer: AdamW

## Resources Used
- Compute: AWS Sagemaker ml.g4dn.xlarge
- Time: About 63 hours


# Evaluation

## BLiMP
Used BLiMP evaluation from the [2024 BabyLM evaluation pipeline repository](https://github.com/babylm/evaluation-pipeline-2024).

### Results
- blimp_supplement accuracy: 47.54%
- blimp_filtered accuracy: 51.79%
- See [blimp_results](./blimp_results) for a detailed breakdown on subtasks.

### Hyperparameters
- Epochs: 1
- Script modified for masked LMs

### Resources Used
- Compute: arm64 MacOS
- Time: About 30 minutes