You can deploy this model and start earning money today!

mesolitica/
Malaysian-Qwen2.5-72B-Instruct

Name: Malaysian-Qwen2.5-72B-Instruct
Availability: InStock

safetensors

qwen2

Views

No views yet

Malaysian Qwen 2.5 72B Instruct

Continue finetuning https://huggingface.co/Qwen/Qwen2.5-72B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.

Improvement

Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.

Training session

Finetune on mesolitica/Malaysian-SFT to make the model understand Malaysian context.

How we train

LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
128 Rank with alpha 256, or alpha of 2.0
Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
Chunk CCE loss for LoRA.
WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-72b-malaysian-8k?nw=nwuserhuseinzol05

Source code at https://github.com/mesolitica/malaya/tree/master/session/qwen2.5

Benchmark

MalayMMLU

Probability next tokens

Based on 0-shot official MalayMMLU First token accuracy,

                            Model   Accuracy   shot by_letter        category
0  Malaysian-Qwen2.5-72B-Instruct  81.620958  0shot      True            STEM
1  Malaysian-Qwen2.5-72B-Instruct  80.820611  0shot      True        Language
2  Malaysian-Qwen2.5-72B-Instruct  77.536860  0shot      True  Social science
3  Malaysian-Qwen2.5-72B-Instruct  76.900935  0shot      True          Others
4  Malaysian-Qwen2.5-72B-Instruct  82.730375  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Qwen2.5-72B-Instruct
Metric : first
Shot : 0shot
average accuracy 79.63490686821129
accuracy for STEM 81.62095783872289
accuracy for Language 80.82061068702289
accuracy for Social science 77.53686036426713
accuracy for Others 76.90093547613337
accuracy for Humanities 82.73037542662117

While the original model,

                  Model   Accuracy   shot by_letter        category
0  Qwen2.5-72B-Instruct  80.884159  0shot      True            STEM
1  Qwen2.5-72B-Instruct  79.103053  0shot      True        Language
2  Qwen2.5-72B-Instruct  75.802255  0shot      True  Social science
3  Qwen2.5-72B-Instruct  75.053970  0shot      True          Others
4  Qwen2.5-72B-Instruct  79.977247  0shot      True      Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Qwen2.5-72B-Instruct
Metric : first
Shot : 0shot
average accuracy 77.80118118366167
accuracy for STEM 80.88415882112157
accuracy for Language 79.1030534351145
accuracy for Social science 75.80225498699046
accuracy for Others 75.05396977692492
accuracy for Humanities 79.97724687144482

First token match using vLLM

Based on 0-shot exact first token match using vLLM Guided Decoding,

                            Model   Accuracy  shot        category
0  Malaysian-Qwen2.5-72B-Instruct  80.229226     0            STEM
1  Malaysian-Qwen2.5-72B-Instruct  78.101145     0        Language
2  Malaysian-Qwen2.5-72B-Instruct  75.252963     0  Social science
3  Malaysian-Qwen2.5-72B-Instruct  74.358359     0          Others
4  Malaysian-Qwen2.5-72B-Instruct  80.477816     0      Humanities
Model : Malaysian-Qwen2.5-72B-Instruct
Metric : full
Shot : 0
average accuracy 77.28905959608475
accuracy for STEM 80.22922636103151
accuracy for Language 78.10114503816794
accuracy for Social science 75.25296328418618
accuracy for Others 74.35835931878148
accuracy for Humanities 80.4778156996587