Beta
Explore
Marketplace
Neural Labs
Playground
Wallet
Docs
Urdu-Bert – AI Model by mahwizzzz | AlphaNeural AI
You can deploy this model and start earning money today!
mahwizzzz
/
Urdu-Bert
like
0
ur
mit
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Urdu-BERT Pretraining (PyTorch)
I implemented BERT model from scratch with Pytorch on Urdu Data. It uses Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) tasks, just like the original BERT model.
✅ Features
Trained on
Urdu-1M-news-text
Custom WordPiece tokenizer
Multi-head attention & transformer encoder blocks
NSP and MLM heads
Uses PyTorch and HuggingFace Tokenizers
Training tracked with Weights & Biases (WandB)
⚙️ Training Setup
Setting
Value
Epochs
20
Batch Size
64
Sequence Length
64 tokens
Embedding Size
128
Encoder Layers
2
Attention Heads
2
Max LR
2.5e-5
Warmup Steps
1000
Optimizer
Adam
📊 Training Loss (WandB)
Below is an example of the loss curve during training:
WandB Loss Graph
This shows MLM loss, NSP loss, and total loss reducing over time.
📌 Notes
NSP uses random sentence pairs for false samples
Positional embeddings are created using sin/cos
Custom learning rate scheduler with warm-up
Model is built completely from scratch (no pretrained weights)
📚 References
BERT from Scratch (Medium)
PyTorch BERT Tutorial
