This model is used for Masked Language Modeling (MLM) and has been trained on the IMDB dataset. In training this model, we did not use whole word masking In our preprocessing on the dataset, we tokenized all the texts and then placed them into a single dictionary. We also used chunking.

Intended uses & limitations

This model is used for the masked word prediction task. An improved version of the model will be added in the coming days

Training and evaluation data

imdb

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 64
eval_batch_size: 64
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 3.0
mixed_precision_training: Native AMP

Training results

Perplexity: 9.51

Framework versions

Transformers 4.51.3
Pytorch 2.6.0+cu124
Datasets 3.5.1
Tokenizers 0.21.1

Alireza0017/distilbert-base-uncased-finetuned-imdb

distilbert-base-uncased-finetuned-imdb

Model description