You can deploy this model and start earning money today!

JUSTIAM/
distilbert-base-uncased-sst2-sentiment-kd

Name: distilbert-base-uncased-sst2-sentiment-kd
Availability: InStock

Views

No views yet

DistilBERT Base Uncased SST-2 Sentiment Model (Distilled from BERT)

This model is a DistilBERT-base-uncased model fine-tuned on the SST-2 (Stanford Sentiment Treebank) dataset using knowledge distillation from a BERT-base teacher (textattack/bert-base-uncased-SST-2).

It achieves 90.83% validation accuracy — just 1.61% below the teacher — while being 1.6× smaller and nearly 2× faster at inference time.

Model Details

Student architecture: distilbert-base-uncased
Teacher model: textattack/bert-base-uncased-SST-2
Distillation method: Logits-based knowledge distillation (Hinton et al., 2015)
Temperature: 4.0
Loss weighting: α = 0.7 (70% distillation loss, 30% hard-label loss)
Training steps: 3 epochs, AdamW (lr=5e-5), batch size=16
Reproducibility: Fixed random seed (42)

Intended Use

This model is intended for binary sentiment classification of English sentences (e.g., movie reviews). It outputs two classes:

0: Negative sentiment
1: Positive sentiment

Out-of-Scope Use

Not suitable for non-English text
Not designed for fine-grained sentiment (e.g., 5-star ratings)
Not evaluated on social media text with heavy slang or emojis

Performance

Model	SST-2 Val Accuracy	Parameters	Speed (samples/sec)
Teacher (BERT-base)	92.43%	109.5M	~1,980
This model (DistilBERT)	90.83%	67.0M	~3,915
Fresh (untrained) DistilBERT	54.47%	67.0M	3,915

Distillation efficiency: 95.77%
Size reduction: 38.8% fewer parameters
Inference speedup: 1.98× faster than teacher

Usage

With Transformers Pipeline

python

from transformers import pipeline

classifier = pipeline(
    "sentiment-analysis",
    model="JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd",
    tokenizer="JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd"
)

result = classifier("This movie is absolutely fantastic!")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.998}]

Manual Inference

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "I really hate this boring film."
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits
    probabilities = torch.softmax(logits, dim=-1)
    predicted_class = torch.argmax(probabilities, dim=-1).item()

sentiment = "POSITIVE" if predicted_class == 1 else "NEGATIVE"
confidence = probabilities[0][predicted_class].item()

print(f"Sentiment: {sentiment}, Confidence: {confidence:.3f}")

Training Data

Dataset: GLUE/SST-2
Task: Binary sentiment classification
Train samples: 67,349
Validation samples: 872

Evaluation

Evaluated on the official SST-2 validation set. No test set used.