---
license: apache-2.0
language:
- en
base_model:
- bsu-slim/electra-tiny
pipeline_tag: text-classification
library_name: transformers
---

A pretrained [ELECTRA-Tiny](https://huggingface.co/bsu-slim/electra-tiny/tree/main) model. Used pretraining [data](https://osf.io/5mk3x) 
from the [2024 BabyLM Challenge](https://babylm.github.io/index.html). Used to perform text classification 
on the [Web of Science Dataset WOS-46985](https://data.mendeley.com/datasets/9rw3vkcfy4/6) but this model is not currently finetuned
for that task. Also evaluated on BLiMP using the [2024 BabyLM evaluation pipeline](https://github.com/babylm/evaluation-pipeline-2024).


# Training
Used pretraining pipeline as defined in this [repository](https://github.com/bakirgrbic/bblm).

## Hyperparameters
- Epochs: 10
- Batch size: 8
- Learning rate: 1e-4
- Optimizer: AdamW

## Resources Used
- Compute: AWS Sagemaker ml.g4dn.xlarge
- Time: About 70 hours or 3 days


# Evaluation

## Web of Science (WOS)
Used WOS pipeline as defined in this [repository](https://github.com/bakirgrbic/bblm).

### Results
- 76% accuracy on the last epoch of the test set.

### Hyperparameters
- Epochs: 3
- Batch size: 64
- Learning rate: 2e-5
- Optimizer: AdamW
- Max Length: 128
- Parameter Freezing: None

### Resources Used
- Compute: AWS Sagemaker ml.g4dn.xlarge
- Time: About 5 minutes


## BLiMP

### Results
- blimp_supplement accuracy: 49.79%
- blimp_filtered accuracy: 50.65%
- See [blimp_results](./blimp_results) for a detailed breakdown on subtasks.

### Hyperparameters
- Epochs: 1
- Script modified for masked LMs

### Resources Used
- Compute: arm64 MacOS
- Time: About 1 hour