You can deploy this model and start earning money today!

Minuri/
sinhala-llama-1b-corpus-diverse

Name: sinhala-llama-1b-corpus-diverse
Availability: InStock

continual-pretraining

text-generation

meta-llama/Llama-3.2-1B

finetune

llama3.2

Views

No views yet

Sinhala LLaMA 1B (Diversity-optimized)

A continually pretrained LLaMA 3.2 1B model on a diversity-optimized Sinhala corpus, with an extended Sinhala tokenizer. This is the best-performing model in the experimental setup, achieving a perplexity of 10.49 on the Sinhala test set - a 99.98% reduction from the base model's perplexity.

Model variants in this series:

Minuri/sinhala-llama-1b-corpus-news - Trained on news-only corpus (Model A) | Perplexity: 14.68

Minuri/sinhala-llama-1b-corpus-random - Trained on random corpus (Model B) | Perplexity: 10.86

Minuri/sinhala-llama-1b-corpus-diverse - Trained on diversity-optimized corpus (Model C) | Perplexity: 10.49 ✅ Best

Model Description

This model is the result of two-stage continual pretraining (CPT) of meta-llama/Llama-3.2-1B on Minuri/sinhala-corpus-c-diverse-1m (1M diversity-optimized Sinhala sentences), using an extended tokenizer (Minuri/sinhala-llama-3.2-1b-tokenizer) that adds 7,843 Sinhala-specific tokens to the base vocabulary.

The diversity-optimized corpus was selected using a combination of semantic diversity (embedding spread via paraphrase-multilingual-mpnet-base-v2) and lexical diversity metrics (TTR, MATTR, hapax ratio) from the 12.38M sentence Minuri/diverse_sinhala_dataset.

Training Details

Parameter	Value
Base model	`meta-llama/Llama-3.2-1B`
Tokenizer	`Minuri/sinhala-llama-3.2-1b-tokenizer`
Training corpus	`Minuri/sinhala-corpus-c-diverse-1m` (1M sentences)
Training approach	Two-stage continual pretraining
Extended vocab size	136,099 tokens
Token reduction on Sinhala	~70.4%

Evaluation Results

Evaluated on Minuri/sinhala-test-set-50k using token-level perplexity. Baseline perplexity is measured using the base LLaMA 3.2 1B model with the extended Sinhala tokenizer - the high baseline reflects the mismatch between the base model weights and the new vocabulary. When evaluated with the original base tokenizer, the base model perplexity is ~2.

Model	Corpus	Baseline PPL	Trained PPL	Improvement	Loss
Model A (`corpus-news`)	News-only	66,448.86	14.67	99.98%	2.686
Model B (`corpus-random`)	Random	66,454.47	10.86	99.98%	2.385
Model C (this repo)	Diversity-optimized	66,439.92	10.49	99.98%	2.351 ✅

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Minuri/sinhala-llama-1b-corpus-diverse")
model = AutoModelForCausalLM.from_pretrained("Minuri/sinhala-llama-1b-corpus-diverse")

inputs = tokenizer("ශ්‍රී ලංකාව", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Uses

Sinhala text generation and language modelling
Base model for Sinhala SFT experiments
Low-resource NLP research and benchmarking

Limitations

1B parameter model with limited reasoning capability without SFT
Primarily optimized for Sinhala; English capability may be reduced relative to the base model
Not instruction-tuned - outputs are continuations, not responses

Related Repositories

Repo	Description
`Minuri/sinhala-llama-3.2-1b-tokenizer`	Extended Sinhala tokenizer
`Minuri/sinhala-corpus-c-diverse-1m`	Training corpus
`Minuri/sinhala-test-set-50k`	Evaluation test set
`Minuri/sinhala-corpus-a-news-1m`	Corpus A - news-only
`Minuri/sinhala-corpus-b-random-1m`	Corpus B - random
`Minuri/diverse_sinhala_dataset`	Full parent corpus

License

This model is derived from meta-llama/Llama-3.2-1B and is subject to the LLaMA 3.2 Community License.