AlexStamp/distilbert-base-uncased-finetuned-imdb-standard

DistilBERT Base Uncased - Domain Adaptation, Fine-Tuned on IMDB Movie Reviews (Standard Subword Masking)

This model is a domain-adapted version of distilbert-base-uncased fine-tuned on the IMDB movie reviews dataset for Masked Language Modeling (MLM).

By adapting the base model to informal, domain-specific film terminology, its language modeling performance improved significantly over the base checkpoint.

It achieves the following results on the evaluation set:

• Loss: 2.2938
• Model Preparation Time: 0.0046

Model Description

• Base Model: distilbert-base-uncased
• Task: Domain Adaptation / Masked Language Modeling (Fill-Mask)
• Dataset: stanfordnlp/imdb (all splits concatenated and chunked into 128-token blocks)
• Objective: Adapt general language representations to informal, domain-specific movie review language and vocabulary co-occurrences.

Performance & Results

• Evaluation Metric: Cross-Entropy Loss & Perplexity
• Pre-training Perplexity: 23.14
• Fine-tuned Perplexity: 9.91

Note: For Masked Language Modeling, the loss evaluated during training is the cross-entropy over masked tokens. A standard metric for measuring domain adaptation is Perplexity, calculated as exp(eval_loss). Lower perplexity indicates that the model is less "surprised" by text from the target domain.

Training Details

• Training Dataset Size: 50,000 sequence chunks (128 max token length, full packing / zero padding)
• Evaluation Dataset Size: 10,000 sequence chunks (Official IMDB test split)
• Hardware: NVIDIA T4 GPU (Google Colab)
• Collator Strategy: Standard Subword Masking (DataCollatorForLanguageModeling, 15% probability for dynamic masking)

Note: A Whole-Word Masking (WWM) Collator variant is available in distilbert-base-uncased-finetuned-imdb-wwm, together with its results and a discussion of the comparison.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• weight_decay: 0.01
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3.0
• precision: FP16 mixed-precision training (fp16=True, native PyTorch AMP)

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time
2.4848	1.0	782	2.3654	0.0046
2.4338	2.0	1564	2.3231	0.0046
2.4176	3.0	2346	2.2983	0.0046

Framework versions

• Transformers 5.13.1
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

Usage

from transformers import pipeline

fill_mask = pipeline("fill-mask", model="AlexStamp/distilbert-base-uncased-finetuned-imdb-standard")

# Example domain completion
result = fill_mask("This movie was completely [MASK] and worth every penny!")
for prediction in result:
    print(f"{prediction['token_str']}: {prediction['score']:.4f}")