Our Llama-3.2-3B-Instruct-uz (experimental) model has been continually pretrained with context length of 2048 tokens, on 1.2B tokens (80% English, 20% Uzbek), then SFT fine-tuned. Our customized tokenizer averages 1.7 tokens per Uzbek word vs. ~3.5 in the original Llama models, meaning 2x faster inference and longer effective context length on Uzbek text. You’ll be able to run this model on just 4 GB of VRAM (with quantization), perfect for small GPUs, edge devices, or even mobile scenarios.
Benchmarks
Model
BLEU Uz→En (Zero_shot)
BLEU En→Uz (Zero_shot)
COMET Uz→En
COMET En→Uz
Uzbek Sentiment Analysis
Uzbek News Classification
MMLU (English) (Zero_shot)
Llama-3.2 1B Instruct
3.62
0.44
56.72
35.52
54.77
42.16
38.15
Llama-3.2 1B Instruct Uz
10.33
5.29
74.39
72.34
65.25
17.14
27.20
Llama-3.2 3B Instruct
11.91
2.54
71.96
55.62
56.01
70.60
52.04
Llama-3.2 3B Instruct Uz
20.47
9.18
83.20
80.71
77.55
41.43
45.91
Llama-3.1 8B Instruct
24.23
8.28
83.12
82.22
69.77
73.63
60.59
The results show that our Uzbek-optimized models consistently outperform their base counterparts in translation benchmarks (BLEU and COMET) on the FLORES+ Uz-En / En-Uz evaluation datasets and sentiment analysis in Uzbek language. Also, on the MMLU benchmark, which measures general language understanding across multiple tasks in English, and News classification tasks, our Uzbek optimized model showed slight decline because of catastrophic forgetting of original English instruction following. (The official Llama model’s MMLU score may differ from our score due to our evaluation method. Refer to the links below to see evaluation details.)
Looking ahead, these models are only experimental checkpoints with a room for improvement. We’re eager to see how these models will contribute to Uzbek open-source and be used by our Uzbek 🇺🇿 community. 🚀
How to use
The Llama-3.2-3B-Instruct-uz model can be used with transformers in the following way. We recommend preprocessing Uzbek input to replace apostrophe (') with sequence (APST) to achieve our model's lower tokenizer fertility.
Use with transformers
python
1import re, torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3import langid
45DEVICE = torch.device("cuda"if torch.cuda.is_available()else"cpu")6DTYPE = torch.bfloat16
7MODEL_ID ="bxod/Llama-3.2-3B-Instruct-uz"8PATTERN =r"[’‘‚‛ʻʼʽʾʿˈˊˋˌˍ'\']"910tok = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)11tok.padding_side ="left"12model = AutoModelForCausalLM.from_pretrained(13 MODEL_ID,14 torch_dtype=DTYPE,15 device_map="auto"16)1718EOT ="<|eot_id|>"19SYSTEM =(20f"{tok.bos_token}<|start_header_id|>system<|end_header_id|>\n"21"You are a helpful assistant<|eot_id|>"22)2324defprompt(user:str)->str:25return(26 SYSTEM +27"<|start_header_id|>user<|end_header_id|>\n"+28f"{user}{EOT}"+29"<|start_header_id|>assistant<|end_header_id|>"30)3132defgenerate(user:str, max_new:int=256)->str:33 lang, confidence = langid.classify(user)34 clean_text = re.sub(PATTERN,"APST", user)if lang !="en"else user
35 enc = tok(prompt(clean_text), return_tensors="pt").to(DEVICE)36 out = model.generate(**enc,37 max_new_tokens=max_new,38 bos_token_id=tok.bos_token_id,39 eos_token_id=tok.convert_tokens_to_ids(EOT),40 pad_token_id=tok.pad_token_id,41 do_sample=False)42 txt = tok.decode(out[0], skip_special_tokens=False)43 txt = txt.split("<|start_header_id|>assistant<|end_header_id|>",1)[1]44return txt.split(EOT,1)[0].replace("APST","'").strip()4546print(generate("Menga Alisher Navoiy haqida aytib ber."))
Information on Evaluation Method
To evaluate on the translation task, we used FLORES+ Uz-En / En-Uz datasets.
We used the following prompt to do zero-shot Uz-En evaluation both for the base model and Uzbek-optimized model (for En-Uz eval, we changed the positions of the words "English" and "Uzbek").
python
1 prompt =f"Input: {clean_text} \n\nYour task is to accurately translate the given Uzbek text into English.\n"2"Output only the English translation, without any additional comments.\n"3"\nPlease translate the following Uzbek text into English."
To assess the model's ability in Uzbek sentiment analysis, we used the risqaliyevds/uzbek-sentiment-analysis dataset (refer to behbudiy/uzbek-sentiment-analysis dataset).
We used the following prompt for the evaluation:
python
1prompt =f'''Input: {clean_text} \n\nGiven the following text, determine the sentiment as either 'Positive' or 'Negative.' Respond with only the word 'Positive' or 'Negative' without any additional text or explanation."
2'''
For Uzbek News Classification, we used risqaliyevds/uzbek-zero-shot-classification dataset and asked the model to predict the category of the news using the following prompt:
python
1prompt =f'''Input: {clean_text}\n\nClassify the given news article in Uzbek.
20 - Siyosat - If the text is about politics.
31 - Iqtisodiyot - If the text is about the economy.
42 - Texnologiya - If the text is about technology.
53 - Sport - If the text is about sports.
64 - Madaniyat - If the text is about culture.
75 - Salomatlik - If the text is about health.
86 - Oila va Jamiyat - If the text is about family and society.
97 - TaAPSTlim - If the text is about education.
108 - Ekologiya - If the text is about ecology.
119 - Xorijiy Yangiliklar - If the text is about foreign news.
1213Print only one digit ID of the corresponding class.
14'''
On MMLU, we performed 0-shot evaluation using the following template and extracted the first token generated by the model for measuring accuracy:
template = "Given the above question and choices, choose the single best answer (A, B, C, or D). Respond with only one letter..
