This model is a sentiment classification model built on top of a Domain-Adaptive Pretrained (DAPT) DistilBERT encoder, further fine-tuned on the IMDb movie reviews sentiment classification task.

The encoder was first adapted to the movie review domain using masked language modeling (MLM) on IMDb reviews, and then a linear classification head was trained for binary sentiment prediction.

The model predicts:

Positive
Negative for English movie reviews.

Model architecture

Base model: DistilBERT DAPT IMDb
Classifier Head: Linear layer on top of the [CLS] token.
Pre-classifier: Not used in this implementation
Labels: 2

Fine tuning details

Dataset: stanford/imdb
Loss: Cross-entropy loss
Tokenization: WordPiece
Dynamic padding
Train samples: 25k
Validation samples: 5k
Test samples: 20k

Intended Uses

Supported Use Cases

Sentiment analysis of movie reviews
Opinion mining on review-style English text
NLP research on domain adaptation

Out-of-Scope Uses

Medical, legal or financial decision-making
Safety-critical systems
Content moderation or profiling of individuals
Non-English or highly technical text

Usage

Loading the model

python

from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file as safe_load_file

REPO_ID = "ayushshah/distilbert-dapt-imdb-sentiment"
hf_hub_download(repo_id=REPO_ID, filename="model.py", local_dir=".")

from model import DistilBERTClassifier, infer_reviews

classifier = DistilBERTClassifier()
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)

model_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")
classifier.load_state_dict(safe_load_file(model_path))
classifier.eval()

Inference

python

# For single review inference
sample_review = "Great movie"
label_class, label, conf = infer_reviews(sample_review, classifier, tokenizer)
print(f"Predicted Sentiment: {label} (Confidence: {conf:.4f})\n")

# For batch inference
reviews = ["I loved this film!", "This was a terrible movie."]
results = infer_reviews(reviews, classifier, tokenizer)
for review, (label_class, label, conf) in zip(reviews, results):
    print(f"Predicted Sentiment: {label} (Confidence: {conf:.4f})\n")

Evaluation

Faster convergance during fine-tuning
DAPT improves accuracy compared to DistilBERT base
Not using a pre_classifier gives marginal better results (~0.5%) with lesser parameters

Limitations

Inherits biases present in IMDb reviews
Performance may degrade on non-review text
Not suitable for sarcasm-heavy or highly nuanced sentiment
Not designed for multilingual sentiment analysis

Ethical Considerations

IMDb reviews may reflect demographic and cultural biases
Outputs should not be used to infer personal traits
The model should not be used for automated moderation or sensitive decision-making

Training Hyperparameters

Encoder LR: 1e-5
Classifier LR: 5e-5
Optimizer: AdamW
Epochs: 3
Batch size: 64
Scheduler: Linear decay (min lr: 1e-8) with 10% warmup
Automatic mixed precision training

Training Stats

Epoch 1: Train Loss: 0.3159, Train Acc: 0.8533 | Val Loss: 0.2046, Val Acc: 0.9160
Epoch 2: Train Loss: 0.1744, Train Acc: 0.9341 | Val Loss: 0.1986, Val Acc: 0.9220
Epoch 3: Train Loss: 0.1400, Train Acc: 0.9507 | Val Loss: 0.2009, Val Acc: 0.9230

License

This model follows the license of the base DistilBERT model. Please refer to the Hugging Face repository for license details.

ayushshah/distilbert-dapt-imdb-sentiment

DistilBERT IMDb DAPT Sentiment Classifier