README.md

---
library_name: transformers
language: en
license: apache-2.0
datasets:
- stanfordnlp/sentiment140
base_model:
- google-bert/bert-base-uncased
---

# Model Card: BERT-Sentiment140

An in-domain BERT-base model, pre-trained from scratch on the Sentiment140 dataset text.

## Model Details

### Description

This model is based on the [BERT base (uncased)](https://huggingface.co/google-bert/bert-base-uncased)
architecture and was pre-trained from scratch (in-domain) using the text in Sentiment140 dataset, excluding its test split.
Only the masked language modeling (MLM) objective was used during pre-training.

- **Developed by:** [Cesar Gonzalez-Gutierrez](https://ceguel.es)
- **Funded by:** [ERC](https://erc.europa.eu)
- **Architecture:** BERT-base
- **Language:** English
- **License:** Apache 2.0
- **Base model:** [BERT base model (uncased)](https://huggingface.co/google-bert/bert-base-uncased)

### Checkpoints

Intermediate checkpoints from the pre-training process are available and can be accessed using specific tags,
which correspond to training epochs and steps:

| Epoch | Step | Tags | |
|---|---|---|---|
| 1 | 15000 | epoch-1 | step-15000 |
| 2 | 30000 | epoch-2 | step-30000 |
| 3 | 45000 | epoch-3 | step-45000 |
| 5 | 75000 | epoch-5 | step-75000 |
| 10 | 150000 | epoch-10 | step-150000 |
| 15 | 225000 | epoch-15 | step-225000 |
| 20 | 300000 | epoch-20 | step-300000 |
| 25 | 375000 | epoch-25 | step-375000 |

To load a model from a specific intermediate checkpoint, use the `revision` parameter with the corresponding tag:
```python
from transformers import AutoModelForMaskedLM

model = AutoModelForMaskedLM.from_pretrained("<model-name>", revision="<checkpoint-tag>")
```

### Sources

- **Paper:** [Information pending]

## Training Details

For more details on the training procedure, please refer to the base model's documentation:
[Training procedure](https://huggingface.co/google-bert/bert-base-uncased#training-procedure).

### Training Data

All texts from Sentiment140 dataset, excluding the test partition.

#### Training Hyperparameters

- **Precision:** fp16
- **Batch size:** 32
- **Gradient accumulation steps:** 3

## Uses

For typical use cases and limitations, please refer to the base model's guidance: 
[Inteded uses & limitations](https://huggingface.co/google-bert/bert-base-uncased#intended-uses--limitations).

## Bias, Risks, and Limitations

This model inherits potential risks and limitations from the base model. Refer to:
[Limitations and bias](https://huggingface.co/google-bert/bert-base-uncased#limitations-and-bias).

## Environmental Impact

- **Hardware Type:** NVIDIA Tesla V100 PCIE 32GB
- **Runtime:** 36.5 h
- **Cluster Provider:** [Artemisa](https://artemisa.ific.uv.es/web/)
- **Compute Region:** EU
- **Carbon Emitted:** 6.79 kg CO2 eq.

## Citation

**BibTeX:**

[More Information Needed]