README.md

---
library_name: transformers
tags:
  - bert
  - text-classification
  - sentiment-analysis
  - domain-adaptation
  - mlm
  - imdb
language:
  - en
license: apache-2.0
datasets:
  - stanfordnlp/imdb
metrics:
  - accuracy
  - f1
pipeline_tag: text-classification
---

# BERT MLM Fine-Tuned — Sentiment Classifier (sm3455/bert-mlm-ft)

A domain-adapted BERT model for binary sentiment classification, trained on the Stanford IMDB dataset using a two-stage approach: Masked Language Modeling (MLM) pretraining on unlabeled reviews followed by supervised fine-tuning for sentiment classification.

---

## Model Details

### Model Description

This model follows a two-stage domain adaptation pipeline:

1. **Stage 1 — MLM Pretraining:** `bert-base-uncased` was further pretrained on 50,000 unlabeled IMDB movie reviews using Masked Language Modeling (MLM) with a masking probability of 15% (via `DataCollatorForLanguageModeling`). This adapts BERT's language representations to the movie review domain before any labeled data is introduced.

2. **Stage 2 — Supervised Fine-Tuning:** The domain-adapted MLM checkpoint was then fine-tuned for binary sentiment classification (positive / negative) on 25,000 labeled IMDB training examples, evaluated on the 25,000-example test set.

- **Developed by:** Sai Bhargav Manginapudi
- **Model type:** BERT (Bidirectional Encoder Representations from Transformers)
- **Base model:** `bert-base-uncased`
- **Language:** English
- **License:** Apache 2.0
- **Fine-tuned from:** `bert-base-uncased` → MLM domain-adapted checkpoint → sequence classifier

---

## Model Sources

- **Repository:** [sm3455/bert-mlm-ft on HuggingFace Hub](https://huggingface.co/sm3455/bert-mlm-ft)
- **Demo:** [Live Gradio Demo](https://huggingface.co/sm3455/bert-mlm-ft) — try it directly in your browser

---

## Uses

### Direct Use

This model can be used out-of-the-box for binary sentiment classification on English text — particularly movie reviews or similar informal/opinion-style text.

```python
from transformers import pipeline

classifier = pipeline("text-classification", model="sm3455/bert-mlm-ft")

result = classifier("The movie started with a banger, first half was really nice and then slowly loses its magic.")
print(result)
# [{'label': 'LABEL_0 or LABEL_1', 'score': 0.XX}]
```

Labels: `LABEL_0` = Negative, `LABEL_1` = Positive

### Downstream Use

The domain-adapted MLM checkpoint (Stage 1) can also be extracted and fine-tuned on other NLP tasks in the movie/entertainment domain — NER, aspect-based sentiment, review summarization, etc.

### Out-of-Scope Use

- Not recommended for formal text, technical documents, or non-English content
- Not suitable for multi-class or multi-label sentiment tasks without further fine-tuning
- Should not be used for high-stakes decisions without human review

---

## Bias, Risks, and Limitations

- Trained exclusively on IMDB movie reviews — performance may degrade on other domains (product reviews, news, social media)
- BERT's 512-token limit truncates long reviews; very long inputs may lose tail context
- The IMDB dataset reflects the biases of online English-speaking movie reviewers and may not generalize across demographics or cultures

### Recommendations

Test on your target domain before deployment. For non-movie-review text, consider further domain adaptation using Stage 1 MLM pretraining on your own unlabeled corpus before fine-tuning.

---

## How to Get Started with the Model

```python
from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="sm3455/bert-mlm-ft",
    tokenizer="sm3455/bert-mlm-ft"
)

# Example
print(classifier("This film was an absolute masterpiece from start to finish."))
print(classifier("Terrible pacing, weak characters, and a predictable ending."))
```

---

## Training Details

### Training Data

**Dataset:** [stanfordnlp/imdb](https://huggingface.co/datasets/stanfordnlp/imdb)

| Split | Size | Use |
|---|---|---|
| `unsupervised` | 50,000 unlabeled reviews | Stage 1 — MLM pretraining |
| `train` | 25,000 labeled reviews | Stage 2 — Classification fine-tuning |
| `test` | 25,000 labeled reviews | Evaluation |

Labels are balanced: 50% positive, 50% negative in both train and test splits.

### Training Procedure

#### Preprocessing

- Tokenizer: `bert-base-uncased` (WordPiece)
- Padding: `max_length` (512 tokens)
- Truncation: enabled
- Text column removed before MLM training; label column retained for classification

#### Stage 1 — MLM Pretraining Hyperparameters

| Parameter | Value |
|---|---|
| Base model | `bert-base-uncased` |
| Dataset | IMDB unsupervised (50K) |
| Epochs | 3 |
| Batch size | 16 per device |
| Gradient accumulation steps | 4 (effective batch = 64) |
| Learning rate | 2e-5 |
| LR scheduler | Linear |
| Warmup steps | 500 |
| Optimizer | AdamW (β1=0.9, β2=0.999, ε=1e-8, weight decay=0.01) |
| MLM probability | 0.20 |
| Precision | FP16 mixed precision |

#### Stage 2 — Classification Fine-Tuning Hyperparameters

| Parameter | Value |
|---|---|
| Base model | MLM checkpoint (Stage 1 output) |
| Dataset | IMDB train (25K) |
| Epochs | 3 |
| Batch size | 16 per device |
| Gradient accumulation steps | 4 (effective batch = 64) |
| Learning rate | 2e-5 |
| LR scheduler | Linear |
| Optimizer | AdamW (β1=0.9, β2=0.999, ε=1e-8, weight decay=0.01) |
| Eval strategy | Every 50 steps |
| Precision | FP16 mixed precision |
| Experiment tracking | Weights & Biases (W&B) |

---

## Evaluation

### Testing Data

IMDB test split — 25,000 labeled English movie reviews, balanced across positive and negative classes.

### Metrics

- **Accuracy** — fraction of correctly classified reviews
- **F1 Score** — harmonic mean of precision and recall (macro)

### Results

| Metric | Score |
|---|---|
| **Accuracy** | **94.4%** |
| **F1 Score** | **0.944** |

The two-stage domain adaptation approach (MLM pretraining → supervised fine-tuning) outperforms direct fine-tuning of vanilla `bert-base-uncased`, demonstrating the value of domain adaptation on unlabeled in-domain data before introducing labels.

---

## Technical Specifications

### Model Architecture

- **Architecture:** BERT (bert-base-uncased) with a sequence classification head (`BertForSequenceClassification`)
- **Parameters:** ~110M (BERT-base)
- **Number of labels:** 2 (binary classification)
- **Objective Stage 1:** Masked Language Modeling (MLM)
- **Objective Stage 2:** Cross-entropy classification loss

### Compute Infrastructure

- **Hardware:** GPU (FP16 mixed precision training)
- **Framework:** PyTorch + HuggingFace Transformers
- **Experiment tracking:** Weights & Biases (W&B)

---

## Live Demo

A Gradio demo is available directly on the HuggingFace model page. Enter any movie review text and the model will return a sentiment prediction with a confidence score.

```python
import gradio as gr
from transformers import pipeline

classifier = pipeline("text-classification", model="sm3455/bert-mlm-ft")

def classify_text(text):
    result = classifier(text)
    label = result[0]['label']
    score = result[0]['score']
    return f"{label} (confidence: {score:.2f})"

iface = gr.Interface(
    fn=classify_text,
    inputs=gr.Textbox(lines=2, placeholder="Enter your movie review here..."),
    outputs="text",
    title="BERT Sentiment Classifier",
    description="Domain-adapted BERT fine-tuned on IMDB for binary sentiment classification."
)

iface.launch()
```

---

## Citation

If you use this model, please cite:

```bibtex
@misc{manginapudi2024bertmlmft,
  author = {Sai Bhargav Manginapudi},
  title = {Domain-Adaptive BERT for Sentiment Classification},
  year = {2024},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/sm3455/bert-mlm-ft}}
}
```

---

## Model Card Author

**Sai Bhargav Manginapudi**
M.S. Computer and Information Science — New Jersey Institute of Technology (Dec 2024)
[LinkedIn](https://www.linkedin.com/in/sai-bhargavmanginapudi) | [GitHub](https://github.com/SaiBhargav05) | saibhargav052000@gmail.com