You can deploy this model and start earning money today!

Ak128umar/
distilbert-base-uncased-finetuned-imdb-accelerate

Name: distilbert-base-uncased-finetuned-imdb-accelerate
Availability: InStock

safetensors

distilbert

Views

No views yet

bert-finetuned-mlm-accelerate

This repository contains a BERT model fine-tuned for Masked Language Modeling (MLM) using the 🤗 Accelerate library for efficient training and evaluation.

🧠 Model Overview

Base model: BERT (pretrained, e.g., bert-base-uncased)
Task: Masked Language Modeling (MLM)
Fine-tuning framework: Hugging Face Transformers + Accelerate
Optimizer: AdamW
Learning rate scheduler: Linear scheduler
Training epochs: 3
Loss metric: Cross-entropy loss over masked tokens
Evaluation metric: Perplexity

⚙️ Training Details

Optimizer used: AdamW from PyTorch.
Learning rate scheduler: Linear decay over total training steps.
Epochs: 3
Sequence length: (e.g., 128 or 512 — fill in based on your setup)
Batch size: (fill in if known)
Mixed precision training with 🤗 Accelerate
Concatenated dataset split into fixed-length chunks to avoid truncation and padding inefficiencies.

📊 Evaluation Results

Epoch	Perplexity
0	12.03
1	11.55
2	11.32

Perplexity was calculated on the validation set after each epoch.

🧪 How to Use

You can load the model for masked language modeling using transformers:

python

from transformers import AutoTokenizer, AutoModelForMaskedLM

model_name = "your-username/bert-finetuned-mlm-accelerate"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

text = "The capital of France is [MASK]."
inputs = tokenizer(text, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

mask_token_index = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero(as_tuple=True)[1]
predicted_token_id = outputs.logits[0, mask_token_index, :].argmax(axis=-1)
print(tokenizer.decode(predicted_token_id))

Ak128umar/distilbert-base-uncased-finetuned-imdb-accelerate