Views
No views yet
| Session | Data | Epochs | PPL | Notes |
|---|---|---|---|---|
| S1 | 500K lines | 5 | 78.10 | from scratch |
| S2 | 1.5M lines | 3 | 41.62 | continued |
| S3 | 87M words | 2 | 28.46 | bf16, cosine LR |
| S4 | 87M words | 3 | 35.16 | grouped context, MLM=0.20 |
| S5 | 87M words | 2 | 29.67 | fine polish, MLM=0.15 |
| S6r | 149M words | 2 | 31.66 | grouping=80, LR=5e-6 |
1from transformers import RobertaForMaskedLM
2import sentencepiece as spm, torch
3import torch.nn.functional as F
4from huggingface_hub import hf_hub_download
5
6REPO = "hellosindh/sindhi-bert-base"
7MASK_ID = 32000
8BOS_ID = 2
9EOS_ID = 3
10
11model = RobertaForMaskedLM.from_pretrained(REPO)
12sp_path = hf_hub_download(REPO, "sindhi_bpe_32k.model")
13sp = spm.SentencePieceProcessor()
14sp.Load(sp_path)