A LoRA fine-tuned version of
meta-llama/Llama-3.2-3B-Instruct for
Mongolian language instruction-following and chat.
This model adapts Llama 3.2 3B Instruct to understand and generate fluent Mongolian text. It was fine-tuned using LoRA (Low-Rank Adaptation) on the
saillab/alpaca-mongolian-cleaned dataset containing ~41,600 Mongolian instruction-following examples.
The base model struggles with Mongolian, producing garbled or incoherent text. After fine-tuning, the model generates fluent, coherent Mongolian responses across a wide range of topics.
1from transformers import AutoTokenizer, AutoModelForCausalLM
2from peft import PeftModel
3import torch
4
5BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
6ADAPTER = "munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian"
7
8tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
9model = AutoModelForCausalLM.from_pretrained(
10 BASE_MODEL, torch_dtype=torch.float16, device_map="auto"
11)
12model = PeftModel.from_pretrained(model, ADAPTER)
13model.eval()
14
15messages = [
16 {"role": "system", "content": "You are a helpful assistant that responds in Mongolian."},
17 {"role": "user", "content": "Монгол улсын нийслэл хаана байдаг вэ?"},
18]
19
20inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", return_dict=True).to(model.device)
21
22with torch.no_grad():
23 output = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, do_sample=True)
24
25response = tokenizer.decode(output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
26print(response)
27# Output: Монгол улсын нийслэл нь Улаанбаатар хот юм.
MM-Eval (
arXiv:2411.09492) is a hierarchical benchmark for evaluating LLMs on Mongolian language tasks across 1,840 items.
1@misc{llama32-3b-mongolian-2026,
2 title={Llama-3.2-3B-Instruct-Mongolian},
3 author={Munkhbayar Batkhuu},
4 year={2026},
5 url={https://huggingface.co/munkhbayar-batkhuu/Llama-3.2-3B-Instruct-Mongolian}
6}
