🎬 IMDb Sentiment Analysis with DistilBERT

A Comparative Study of Embedding-Based and Transformer-Based Architectures for IMDb Sentiment Analysis

This repository contains a fine-tuned DistilBERT model for binary sentiment classification (Positive/Negative) of movie reviews. The model was developed as part of a comparative study evaluating simple neural networks, recurrent networks (BiLSTM), and transformer-based architectures on the IMDB dataset.


πŸ‘€ Author

Samad Sohrab


πŸ“Œ Model Summary

  • Base Model: distilbert-base-uncased
  • Task: Binary Text Classification (Sentiment Analysis)
  • Dataset: IMDB Movie Reviews
  • Test Accuracy: 89.95%
  • Framework: PyTorch + Hugging Face Transformers

πŸ—οΈ Model Architecture

The model is built on top of distilbert-base-uncased with a classification head:

Component Details
Base Transformer DistilBERT (6 layers, 768 hidden size, 12 attention heads)
Classifier Head Linear layer (768 β†’ 2) with Dropout
Max Token Length 256
Output Binary labels (LABEL_0 = Negative, LABEL_1 = Positive)

πŸ“Š Performance Comparison

The following table compares the proposed DistilBERT model with baseline architectures evaluated in the same study:

Model Architecture Test Accuracy Final Validation Loss
Baseline Embedding + GlobalAveragePooling + Dense 86.04% 0.323
BiLSTM Embedding + Bidirectional LSTM + Dense 85.44% 0.328
DistilBERT (This Model) Fine-tuned Transformer 89.95% 0.336

Key Finding: While recurrent and simple feedforward networks struggle to generalize on IMDB, the transformer-based DistilBERT achieves the highest accuracy, demonstrating the effectiveness of pre-trained language representations.


πŸš€ How to Use

1. Install Dependencies

pip install transformers torch

2. Load the Model and Predict

from transformers import pipeline classifier = pipeline("text-classification", model="smd20/imdb-distilbert-sentiment") result = classifier("This movie was absolutely fantastic!") print(result)

3. Using PyTorch Directly

import torch from transformers import AutoTokenizer, AutoModelForSequenceClassification model_name = "smd20/imdb-distilbert-sentiment" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) text = "An unforgettable cinematic experience." inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=256) with torch.no_grad(): logits = model(**inputs).logits predicted_class = torch.argmax(logits, dim=-1).item() print(f"Predicted Label: {'Positive' if predicted_class == 1 else 'Negative'}")


πŸ‹οΈ Training Procedure

Hyperparameters

Parameter Value
Epochs 2
Batch Size 16
Learning Rate 5e-5 (AdamW)
Max Sequence Length 256
Training Samples 10,000
Validation Split 10,000 (from test set used as validation)

Training Results

Epoch Training Loss Validation Loss Validation Accuracy
1 0.3598 0.2688 89.58%
2 0.1919 0.3362 89.95%

Note: The model shows slight overfitting after epoch 1, suggesting that training for 1 epoch may be sufficient for optimal generalization.


⚠️ Limitations & Bias

  • Language: The model is trained on English text only. It may not perform well on non-English reviews.
  • Domain: The model is specialized for movie reviews. Performance may degrade on other domains (e.g., product reviews, social media posts).
  • Length: Reviews longer than 256 tokens are truncated, which may lose important context.
  • Sarcasm & Irony: The model may struggle with sarcastic or ironic statements, as is common in sentiment analysis.

πŸ“š Citation

If you use this model in your work, please cite:

@misc{sohrab2026imdb, author = {Samad Sohrab}, title = {IMDb Sentiment Analysis with DistilBERT}, year = {2026}, publisher = {Hugging Face}, howpublished = {https://huggingface.co/smd20/imdb-distilbert-sentiment} }


πŸ“„ License

This model is released under the Apache 2.0 license.

Downloads last month
-
Safetensors
Model size
67M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for smd20/imdb-distilbert-sentiment

Finetuned
(12,902)
this model

Dataset used to train smd20/imdb-distilbert-sentiment