TinyBERT is 7.5 times smaller and 9.4 times faster on inference compared to its teacher BERT model (while DistilBERT is 40% smaller and 1.6 times faster than BERT). The model has been trained on 89_754 examples split into train, validation and test. Each label was perfectly balanced in each split.

Intended uses & limitations

This model is not as accurate as the distilbert-emotion-balanced one because the focus was on speed, which can lead to misinterpretation of complex sentences. Despite this, its performance is quite good and should be more than sufficient for most use cases.

Usage:

python

from transformers import pipeline

# Create the pipeline
emotion_classifier = pipeline('text-classification', model='AdamCodd/tinybert-emotion-balanced')

# Now you can use the pipeline to classify emotions
result = emotion_classifier("We are delighted that you will be coming to visit us. It will be so nice to have you here.")
print(result)
#[{'label': 'joy', 'score': 0.9895486831665039}]

This model faces challenges in accurately categorizing negative sentences, as well as those containing elements of sarcasm or irony. These limitations are largely attributable to TinyBERT's constrained capabilities in semantic understanding. Although the model is generally proficient in emotion detection tasks, it may lack the nuance necessary for interpreting complex emotional nuances.

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 3e-05
train_batch_size: 32
eval_batch_size: 64
seed: 1270
optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 150
num_epochs: 10
weight_decay: 0.01

Training results

          precision    recall  f1-score   support

 sadness     0.9733    0.9245    0.9482      1496
     joy     0.9651    0.8864    0.9240      1496
    love     0.9127    0.9786    0.9445      1496
   anger     0.9479    0.9365    0.9422      1496
    fear     0.9213    0.9004    0.9108      1496
surprise     0.9016    0.9866    0.9422      1496

accuracy                         0.9355      8976
macro avg    0.9370    0.9355    0.9353      8976
weighted avg 0.9370    0.9355    0.9353      8976

test_acc:     0.9354946613311768
test_loss:    0.1809326708316803

Framework versions

Transformers 4.33.0
Pytorch lightning 2.0.8
Tokenizers 0.13.3

If you want to support me, you can here.