The IMDb dataset is tokenized, and words are masked with 0.15 probability with the DataCollatorForLanguageModeling collator. The resulting dataset is downsampled, resulting in 10,000 training samples and 1,000 validation samples.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 64
eval_batch_size: 64
seed: 42
optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: linear
num_epochs: 5
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Model Preparation Time
2.6824	1.0	157	2.5021	0.0018
2.58	2.0	314	2.4416	0.0018
2.5117	3.0	471	2.4641	0.0018
2.5013	4.0	628	2.3796	0.0018
2.4835	5.0	785	2.4220	0.0018

Framework versions

Transformers 4.50.0
Pytorch 2.6.0+cu124
Datasets 3.4.1
Tokenizers 0.21.1

srvmishra832/IMDb_data_subset-MLM_with-inbuilt_collator-distilbert-base-uncased

IMDb_data_subset-MLM_with-inbuilt_collator-distilbert-base-uncased

Model description

Intended uses & limitations

Training and evaluation data

Training procedure

Training hyperparameters

Training results

Framework versions

srvmishra832/
IMDb_data_subset-MLM_with-inbuilt_collator-distilbert-base-uncased