A transformer-based language model pretrained from scratch on a cleaned and deduplicated Telugu corpus. It is trained on high-quality, natural Telugu text collected from diverse sources.
The training corpus was carefully prepared using the following steps to ensure data quality, linguistic relevance, and uniqueness:
1. Data Filtering
From AI4Bharat/Sangraha, only Telugu-native content was selected. Synthetic dataset was excluded.
From allenai/c4 and oscar, only documents identified as Telugu language were retained.
2. Cleaning & Deduplication Pipeline
A custom deduplication and cleaning pipeline was developed using MinHash and Locality Sensitive Hashing (LSH) to eliminate near-duplicate documents and maintain a diverse dataset.
Steps included:
Text Normalization:
Stripping extra whitespaces.
Replacing multiple newlines and tabs with a single space.
MinHash-based Deduplication:
A MinHashLSH index was used with:
num_perm = 128
similarity_threshold = 0.95
Each document was tokenized at the word level and hashed.
Duplicates were detected and removed without adding them to the final corpus.
Model Parameters
The Sakhi model was trained from scratch with the following configuration:
The model was pretrained for 100 hours on 4× A100 GPUs provided by Lambda. Pretraining was done using PyTorch with mixed precision and DDP (DistributedDataParallel) for efficient scaling.
