Preview Release: This is an early preview (v0.1) for validation purposes. Not intended for production use. Evaluation and model quality may improve in future versions.

Gemma-3-Tonsai-1B is a Khmer-enhanced language model built through Continued Pre-Training (CPT) of Google's Gemma 3 1B on a mixture of Khmer, English, and parallel data.

"Tonsai" (ទន្សាយ) means "rabbit" in Khmer.

Note: This is a base model trained via Continued Pre-Training. It is designed as a foundation for downstream task-specific fine-tuning (e.g., translation, summarization, question answering). For best results, we recommend fine-tuning on your target task using Supervised Fine-Tuning (SFT) before deployment.

Model Details


Base Model	google/gemma-3-1b-pt
Training Method	Continued Pre-Training (CPT), full parameter update
Languages	Khmer (km), English (en)
Parameters	~1B
Context Length	4096 tokens
Precision	bfloat16
License	Gemma Terms of Use

Model Lineage

google/gemma-3-1b-pt
  └── mengsay/Gemma-3-Tonsai-1B-v0.1 (CPT on Khmer data)

Training

Data Mix

Dataset	Type	Weight	Role
CulturaX (km)	Monolingual	55%	Khmer web text
Wikipedia (km)	Monolingual	5%	High-quality Khmer
CulturaX (en)	Monolingual	10%	English retention
OPUS-100 (en-km)	Parallel	15%	Cross-lingual alignment
OpenHermes 2.5	Instruction	10%	Instruction following
Khmer Dictionary 44K	Dictionary	5%	Vocabulary knowledge

Hyperparameters

Parameter	Value
Effective batch size	64 (32 per device x 2 grad accum)
Max sequence length	4096
Learning rate	5e-5 (embedding: 1e-5)
LR scheduler	Cosine with warmup
Warmup steps	200
Weight decay	0.01
Optimizer	AdamW 8-bit
Gradient checkpointing	Unsloth
Hardware	NVIDIA RTX PRO 6000 Blackwell (95GB VRAM)

Evaluation

Evaluation on OPUS-100 (en-km) translation and Khmer perplexity tasks.

Perplexity (lower is better)

Dataset	Gemma-3-1B-PT (base)	Tonsai-1B v0.1
Wikipedia (km)	9.06	2.14
CulturaX (km)	7.09	7.90

Khmer Wikipedia perplexity drops dramatically (9.06 → 2.14), showing significant improvement in Khmer text prediction. CulturaX perplexity is comparable, as the model is still mid-training.

Translation (OPUS-100, 500 samples)

Task	Setting	Metric	Gemma-3-1B-PT (base)	Tonsai-1B v0.1
en→km	zero-shot	BLEU	1.62	18.04
en→km	5-shot	BLEU	3.71	19.34
en→km	zero-shot	chrF	4.45	36.25
en→km	5-shot	chrF	16.60	37.14
km→en	zero-shot	BLEU	9.38	19.66
km→en	5-shot	BLEU	13.12	19.00
km→en	zero-shot	chrF	31.21	44.57
km→en	5-shot	chrF	35.70	42.09

Translation performance improves substantially in both directions, especially en→km zero-shot (BLEU 1.62 → 18.04).

Usage

Text Generation

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "mengsay/Gemma-3-Tonsai-1B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "ជីវិតរស់នៅក្នុងទីក្រុងសព្វថ្ងៃពិតជា"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Translation Example

python

prompt = "English: Cambodia is a country in Southeast Asia.\nKhmer:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

With vLLM Serving

bash

# Start vLLM server
python -m vllm.entrypoints.openai.api_server \
    --model mengsay/Gemma-3-Tonsai-1B-v0.1 --port 8000

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.completions.create(
    model="mengsay/Gemma-3-Tonsai-1B-v0.1",
    prompt="Cambodia is",
    max_tokens=200,
)
print(response.choices[0].text)

Intended Use

This model is a continual pre-trained base model — it has been trained to improve Khmer language understanding and generation but has not been fine-tuned for any specific task or instruction following.

Recommended workflow:

Use this model as a starting point for Supervised Fine-Tuning (SFT) on your target task
Example downstream tasks: translation (en↔km), summarization, question answering, text classification
Fine-tuning with even a few thousand task-specific examples can significantly improve performance

Not recommended for:

Direct use as a chatbot or instruction-following assistant (use an instruction-tuned variant instead)
Production deployment without task-specific fine-tuning and evaluation

Limitations

This is a preview release (v0.1) intended for validation and research
This is a CPT base model — fine-tuning on a specific task is recommended before use
Not optimized for instruction following or conversational use
May generate incorrect, biased, or harmful content
Khmer language quality is preliminary; comprehensive benchmarks will follow in future versions
Training data may contain biases present in web-crawled corpora

Citation

bibtex

@misc{tonsai-lm-2026,
  title   = {Tonsai LM: Continued Pre-Training for Khmer Language Models},
  author  = {Mengsay Loem},
  year    = {2026},
  url     = {https://huggingface.co/mengsay/Gemma-3-Tonsai-1B-v0.1}
}

Acknowledgements

Google Gemma team for the base model
Unsloth for training optimization
HuggingFace dataset contributors for open Khmer language resources
Tonsai LM project