distilbert-base-uncased-finetuned-imdb-whole-word-mask
This model is a fine-tuned version of
distilbert-base-uncased on an imdb dataset.
Model description
🟢 Fine-tuning DistilBERT on IMDB for Masked Language Modeling
In this project, you fine-tuned the DistilBERT model on the IMDB dataset for the task of Masked Language Modeling (MLM). MLM is a self-supervised learning objective where some tokens in a sentence are randomly masked, and the model learns to predict them using context.
🔧 Project Details:
Base model: DistilBERT (a lighter, faster version of BERT)
Training objective: Masked Language Modeling (MLM)
Masking strategy: Whole Word Masking (WWM) used for better context learning
Dataset: IMDB movie reviews
✅ Advantages:
Fine-tuning helps adapt the model to domain-specific language (e.g., movie reviews).
Whole Word Masking helps the model learn more meaningful semantic patterns by masking entire words.
Intended uses & limitations
⚠️ Limitations:
IMDB is originally meant for sentiment analysis, not pretraining, so its linguistic diversity is limited.
DistilBERT is a compressed version of BERT and lacks some layers — so its performance might be lower on complex tasks.
Improper settings like chunk_size or wwm_probability can lead to underfitting or unstable training.
MLM alone is not sufficient for downstream tasks — further fine-tuning is needed for classification, QA, etc.
Training and evaluation data
🟢 Training and Evaluation Data Explanation
In this project, both training and evaluation data were derived from the IMDB dataset, which contains user-written movie reviews. While it's primarily designed for sentiment analysis, we repurposed it for Masked Language Modeling (MLM).
🔹 Training:
The dataset was tokenized using a tokenizer compatible with DistilBERT.
Then, we applied Whole Word Masking (WWM), randomly masking full words instead of individual subwords.
The text was divided into chunks of fixed length (e.g., 128 or 256 tokens), which were used as inputs to the model.
The model was trained to predict the masked tokens, learning the language structure of movie reviews.
🔹 Evaluation:
A held-out portion of the IMDB dataset (validation or test split) was used for evaluation.
The same masking and chunking procedures were applied to ensure consistency.
Metrics such as loss and perplexity were computed to assess how well the model predicted the masked tokens.
⚠️ Important Note:
Since IMDB is not a traditional pretraining dataset (like Wikipedia or BookCorpus), its linguistic coverage is limited.
Still, it helps the model adapt to domain-specific patterns of movie review language.
Training procedure
🟢 Training Procedure
In this project, the DistilBERT model was fine-tuned on the IMDB movie review dataset using the Masked Language Modeling (MLM) objective.
🔧 Step-by-step process:
Tokenization:
Each text input was tokenized using DistilBERT’s tokenizer.
Texts were split into fixed-length chunks (e.g., 128 or 256 tokens) to standardize inputs.
Whole Word Masking (WWM):
With a probability of 20%, entire words were randomly masked.
This teaches the model to predict full masked words based on surrounding context.
Training Setup:
Used AdamW optimizer and standard learning rate schedules.
Trained for a number of epochs based on dataset size and loss convergence.
fp16 (mixed precision training) was enabled for faster training.
Training loss was logged at every logging_step to monitor progress.
Evaluation:
After each epoch, the model was evaluated on a validation split.
Metrics like perplexity were used to measure how well the model understood and predicted masked words.
Model Saving:
The fine-tuned model was saved and ready for downstream tasks like sentiment classification or question answering.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- num_epochs: 3.0
- mixed_precision_training: Native AMP
Training results
Framework versions
- Transformers 4.51.3
- Pytorch 2.6.0+cu124
- Datasets 3.5.1
- Tokenizers 0.21.1
