README.md

---
library_name: transformers
tags:
- text-classification
- sentiment-analysis
- lora
- peft
- distilbert
- imdb
---

# Model Card for distilbert_cls-lora-IMDB

## Model Details

### Model Description

This model is a **LoRA-adapted DistilBERT model** fine-tuned for **binary sentiment classification** (POSITIVE / NEGATIVE) on the **IMDB movie reviews dataset**.

Instead of fine-tuning all parameters, **Low-Rank Adaptation (LoRA)** was applied to the attention projection layers, enabling efficient training with a small number of trainable parameters while preserving the original pretrained weights.

⚠️ **Note:** This repository contains **LoRA adapter weights only**, not a fully merged model.

- **Developed by:** Chetan Fernandis
- **Model type:** Transformer encoder (DistilBERT) + LoRA adapters
- **Task:** Sentiment Classification (Binary)
- **Language(s):** English
- **License:** Apache-2.0
- **Finetuned from:** `distilbert-base-uncased`

---

## Model Sources

- **Base Model:** https://huggingface.co/distilbert-base-uncased
- **Dataset:** https://huggingface.co/datasets/imdb
- **Repository:** https://huggingface.co/ChetanFernandis/distilbert_cls-lora-IMDB

## Evaluation Results

The model was evaluated on a held-out validation subset of the **IMDB dataset** using standard classification metrics.

### Confusion Matrix

- **NEGATIVE → NEGATIVE:** 42  
- **NEGATIVE → POSITIVE:** 11  
- **POSITIVE → NEGATIVE:** 7  
- **POSITIVE → POSITIVE:** 40  

This indicates balanced performance across both sentiment classes.


### Classification Report

    

![image](https://cdn-uploads.huggingface.co/production/uploads/6864251ed4dde09fbaabf6cf/sFiCggbBtFF9zYZeAJ1Wo.png)

### Summary

- **Overall Accuracy:** 82%
- **Balanced F1-score:** 0.82 for both classes
- Strong precision for **NEGATIVE** reviews
- Strong recall for **POSITIVE** reviews

These results demonstrate that **LoRA fine-tuning** achieves competitive sentiment classification performance while training only a small fraction of model parameters.

![image](https://cdn-uploads.huggingface.co/production/uploads/6864251ed4dde09fbaabf6cf/1lZVhPe2w0Y61MGhGBPMU.png)

## Uses

### Direct Use

This model can be used for **sentiment analysis** on English text, classifying input sentences or paragraphs as:

- `POSITIVE`
- `NEGATIVE`

Example use cases:
- Movie review analysis
- User feedback classification
- Opinion mining

---

### Out-of-Scope Use

- Not suitable for multilingual sentiment analysis
- Not intended for fine-grained sentiment (e.g., star ratings)
- Not designed for long documents beyond 512 tokens

---

## Bias, Risks, and Limitations

- The model inherits biases from the **IMDB dataset** and the **DistilBERT pretraining corpus**
- Performance may degrade on:
  - Informal language
  - Sarcasm
  - Domain-specific jargon
- Predictions should not be used for high-stakes decisions without human review

---

## How to Get Started with the Model

⚠️ This is a **LoRA adapter**, so it must be loaded on top of the base model.

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=2
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "ChetanFernandis/distilbert_cls-lora-IMDB"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "ChetanFernandis/distilbert_cls-lora-IMDB"
)

# Inference
text = "This movie was absolutely fantastic!"
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)

outputs = model(**inputs)
prediction = outputs.logits.argmax(dim=-1).item()

label_map = {0: "NEGATIVE", 1: "POSITIVE"}
print(label_map[prediction])

---

## Training Details
Training Data

Dataset: IMDB Movie Reviews

Samples: 200 training / 100 test

train_ds = imdb_dataset['train'].shuffle(seed=42).select(range(200))
val_ds   = imdb_dataset['test'].shuffle(seed=42).select(range(100))


Labels: Binary (Positive / Negative)

Training Procedure
Preprocessing
 a. Text tokenized using AutoTokenizer
 b.Truncation applied to max sequence length
 c. Padding applied dynamically per batch

Training Hyperparameters:-
  a. Training regime: FP32 (full precision)
  b. Batch size: 8
  c. Gradient accumulation: 4
  d. Epochs: 20
  e. Optimizer: AdamW
  f. LoRA rank (r): 4
  g. LoRA alpha: 8
  h. LoRA dropout: 0.1
  i. Target modules: q_lin, k_lin, v_lin

Speeds, Sizes, Times:-
  a. Trainable parameters: ~700K
  b. Total parameters: ~67M
  c. Trainable %: ~1%
  d. Checkpoint size: ~3–4 MB (adapter only)

Evaluation:-
 Metrics
  a. Accuracy
  b. Loss (Cross-Entropy)

Results:-
The LoRA-adapted model achieves competitive sentiment classification performance compared to full fine-tuning, while significantly reducing memory usage and training cost.

Technical Specifications
Model Architecture
 a. Base architecture: DistilBERT (6 layers, 12 heads)
 b. Hidden size: 768
 c.LoRA injected into: Attention Q, K, V projections
 d. Classification head: 2-class linear classifier

Compute Infrastructure
 a. Hardware
 b. CPU-based training (no GPU required)

Software
 a.transformers
 b.peft
 c. torch
 d.datasets

Citation

If you use this model, please cite the base model and dataset:

@misc{sanh2019distilbert,
  title={DistilBERT, a distilled version of BERT},
  author={Sanh, Victor and others},
  year={2019},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}