Views
No views yet
| Model | Tokenizer | Vocab | IndicGLUE (Avg) | MultiBLiMP (Avg) | Perplexity |
|---|---|---|---|---|---|
| gpt_10M_bpe_32k | BPE | 32K | 60.45% | 87.69% | 129.25 |
| gpt_10M_character_bigram | Char-Bigram | 2K | 59.94% | 88.32% | 6.03 |
| gpt_10M_sentencepiece_unigram | SP Unigram | 32K | 59.82% | 87.29% | 124.56 |
| gpt_10M_bpe_8k | BPE | 8K | 58.53% | 86.51% | 84.15 |
| gpt_10M_wordpiece | WordPiece | 32K | 58.20% | 86.14% | 124.70 |
| gpt_10M_bpe_16k | BPE | 16K | 57.62% | 87.96% | 113.17 |
| gpt_10M_character_level | Char-Level | 200 | 53.84% | 18.32% | 3.93 |
| Model | Tokenizer | Vocab | IndicGLUE (Avg) | MultiBLiMP (Avg) | Perplexity |
|---|---|---|---|---|---|
| gpt_100M_large | BPE | 32K | 62.09% | 92.54% | 83.50 |
| Model | Tokenizer | Vocab | IndicGLUE (Avg) | MultiBLiMP (Avg) | Perplexity |
|---|---|---|---|---|---|
| deberta_10M_bpe_32K | BPE | 32K | 47.70% | 69.74% | 616.69 |
| deberta_10M_wordpiece_32K | WordPiece | 32K | 40.92% | 69.69% | 525.93 |
| deberta_10M_sentencepiece_unigram_32K | SP Unigram | 32K | 38.93% | 68.51% | 644.42 |
Note: Perplexity values are not directly comparable across tokenizers with different vocabulary sizes, as the prediction space differs. Character-level models have naturally lower perplexity due to smaller vocabularies.
| Task | Accuracy |
|---|---|
| BBC Articles Classification | 78.06% |
| Product Review Sentiment | 73.42% |
| Discourse Mode | 73.32% |
| Choice of Plausible Alternatives | 63.64% |
| Movie Review Sentiment | 61.94% |
| Wikipedia Section Title Prediction | 43.70% |
| Cloze-style Multiple-Choice QA | 40.70% |
| Average (7 tasks) | 62.09% |
WinogradNLI was skipped as the dataset contains only the entailment class in the train/validation splits.
| Phenomenon | Accuracy | Correct / Total |
|---|---|---|
| Subject-Verb Agreement: Person (SV-P) | 96.60% | 398 / 412 |
| Subject-Predicate Agreement: Number (SP-#) | 95.00% | 95 / 100 |
| Subject-Predicate Agreement: Gender (SP-G) | 92.66% | 101 / 109 |
| Subject-Verb Agreement: Gender (SV-G) | 90.21% | 378 / 419 |
| Subject-Verb Agreement: Number (SV-#) | 88.21% | 359 / 407 |
| Overall (1,447 pairs) | 92.54% | — |
1import torch
2from transformers import GPT2LMHeadModel, PreTrainedTokenizerFast
3
4# Load model and tokenizer
5model = GPT2LMHeadModel.from_pretrained("Ayush-Talreja/hindi-babylm")
6tokenizer = PreTrainedTokenizerFast.from_pretrained("Ayush-Talreja/hindi-babylm")
7
8# Generate text
9input_text = "भारत एक"
10input_ids = tokenizer.encode(input_text, return_tensors="pt")
11
12with torch.no_grad():
13 output = model.generate(
14 input_ids,
15 max_new_tokens=50,
16 do_sample=True,
17 top_k=50,
18 top_p=0.95,
19 temperature=0.8,
20 )
21
22generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
23print(generated_text)| Split | Words | Documents |
|---|---|---|
| Training | 100M | 113,266 |
| Validation | 10M | 180,259 |
| Test | 10M | 180,399 |
| Source | Proportion | Description |
|---|---|---|
| IndicCorp V2 | ~50% | Curated news and general web text |
| Hindi Wikipedia | ~30% | Encyclopedia and reference material |
| IndicDialogue | ~15% | Movie and TV show subtitles |
| Children's Literature | ~5% | Stories and educational content |
| Parameter | Value |
|---|---|
| Optimizer | AdamW (β₁=0.9, β₂=0.999) |
| Learning Rate | 3e-4 |
| LR Schedule | Cosine with warmup |
| Batch Size | 32 × 8 gradient accumulation = 256 effective |
| Epochs | 10 |
| Weight Decay | 0.01 |
| Gradient Clipping | max_norm = 1.0 |
| Mixed Precision | BF16 |
| Hardware | NVIDIA GPU (LRZ HPC cluster) |
1@misc{talreja2025hindibabylm,
2 title={Hindi BabyLM: Data-Efficient Language Modeling for Hindi},
3 author={Talreja, Ayush},
4 year={2025},
5 howpublished={\url{https://huggingface.co/Ayush-Talreja/hindi-babylm}},
6 note={BabyLM Challenge adaptation for morphologically rich languages}
7}