MaithiliCLM: First Causal Language Model for the Maithili Language
MaithiliCLM is a decoder-only causal language model (109.5M parameters) pretrained from scratch on MaithiliCorpus — the largest open Maithili text corpus comprising 39.8M clean, LID-filtered, and deduplicated words. This is the first causal/generative language model ever trained specifically for Maithili, a low-resource Indo-Aryan language spoken by approximately 50 million people primarily in Bihar, India and Nepal.
Unlike encoder-only models (BERT, RoBERTa) that only support understanding tasks, MaithiliCLM enables text generation, making it suitable for creative writing, dialogue systems, summarization, and other generative applications in Maithili.
Model Architecture
| Property | Value |
|---|
| Architecture | Decoder-only Transformer (GPT-family) |
| Parameters | 109,545,984 |
| Vocabulary | Byte-level BPE, 32,000 tokens |
| Context Window | 1,024 tokens |
| Hidden Size | 768 |
| Intermediate Size | 3,072 |
| Attention Heads | 12 |
| Hidden Layers | 12 |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Position Encoding | Rotary Position Embeddings (RoPE) |
| Dropout | 0.1 |
| Training Data | MaithiliCorpus LID-filtered (48,064 documents, 27.5M words) |
| Training Tokens | 983M |
| Training Steps | 30,000 |
| Effective Batch Size | 32 |
| Optimizer | AdamW (lr=3e-4, weight_decay=0.1) |
| LR Schedule | Cosine with warmup |
| Precision | FP16 mixed |
| License | CC-BY-4.0 |
Architecture Details
MaithiliCLM implements a modernized GPT-2 family architecture with the following enhancements:
- Rotary Position Embeddings (RoPE): Enables better length generalization than learned absolute positional embeddings
- RMSNorm: More efficient than LayerNorm, used in LLaMA and recent models
- SwiGLU: Improved activation function over GeLU, used in PaLM and LLaMA
- Causal Self-Attention: Standard autoregressive masking for language modeling
Tokenizer
The tokenizer is a byte-level BPE trained from scratch on MaithiliCorpus with a vocabulary of 32,000 tokens. This compact vocabulary reduces embedding parameters from 153.6M to 24.6M, shrinking the embedding layer from 55% to 22% of total model parameters.
| Property | Value |
|---|
| Algorithm | Byte-level BPE |
| Vocabulary Size | 32,000 |
| Training Data | MaithiliCorpus (39.8M words) |
| Tokens per Word | 3.53 |
| Special Tokens | <s>, </s>, <unk>, <pad> |
Training
Data Preparation
The training corpus underwent rigorous cleaning and quality filtering:
- Source: AiventraLab/maithili-corpus (49,629 raw documents)
- Cleaning: HTML removal, encoding normalization, language verification
- LID Filtering: Retained documents with ≥50% Maithili sentences (48,064 documents)
- Train/Val Split: 47,103 training / 961 validation documents
- Tokenization: Byte-level BPE encoding, packed into 1,024-token blocks
Training Configuration
- Duration: ~12 hours
- Tokens Seen: 983M (~10.3 epochs)
- Peak VRAM: 6.46 GB
- Throughput: ~26,500 tokens/second
Training Dynamics
| Metric | Step 0 | Step 10K | Step 20K | Step 30K |
|---|
| Train Loss | 10.32 | 3.65 | 3.18 | 3.04 |
| Val Loss | 10.31 | 3.42 | 3.18 | 3.12 |
| Val PPL | 29,674 | 30.56 | 24.05 | 22.68 |
Training plateaued after ~30K steps with diminishing returns (<1 ppl improvement per 5K steps).
Generation Quality
The model generates coherent Maithili text across diverse prompts. Generated continuations are syntactically correct and semantically plausible, though coherence degrades with longer sequences due to the limited training data.
Intended Use
- Text Generation: Creative writing, article completion, dialogue systems
- Fine-tuning Base: Adapt for classification, NER, QA, summarization
- Research: Foundation for Maithili NLP, low-resource language modeling
- Education: Teaching Maithili language and computational linguistics
How to Use
1from transformers import AutoTokenizer
2import torch
3
4# Load tokenizer
5tokenizer = AutoTokenizer.from_pretrained("AiventraLab/maithili-clm")
6
7# Load model (requires custom model class - see model.py)
8# model = MaithiliGPT.from_pretrained("AiventraLab/maithili-clm")
9
10# Generate text
11prompt = "मैथिली भाषा एक"
12input_ids = tokenizer.encode(prompt, return_tensors="pt")
13
14# Autoregressive generation
15output = model.generate(
16 input_ids,
17 max_new_tokens=100,
18 temperature=0.8,
19 top_k=50,
20 top_p=0.95,
21 do_sample=True
22)
23
24generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
25print(generated_text)
Acknowledgments
The authors gratefully acknowledge
AnK Komputing for providing the computational resources and infrastructure used to conduct the simulations/analyses presented in this work. We thank the team at AnK Komputing for their technical support and access to their high-performance computing services.
Citation
1@article{maithili2026clm,
2 title={MaithiliCLM: First Causal Language Model for the Maithili Language},
3 author={[Authors]},
4 journal={arXiv preprint},
5 year={2026}
6}
License
CC-BY 4.0 applies to the model weights, tokenizer, and documentation. The training data (MaithiliCorpus) retains its original license terms.
See Also
- MaithiliCorpus - Training data
- MaithiliBERT - Encoder-only counterpart
