Malaysian Qwen 2.5 7B Instruct
Continue finetuning
https://huggingface.co/Qwen/Qwen2.5-7B-Instruct on highly curated 1.5B tokens Malaysian instruction dataset.
We provide 2 different revisions,
Rank 128, Alpha 256, 83a0e145c726385502898ab7e016982eae1b684d
Rank 256, Alpha 512, 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1
Improvement
Support respond in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Able to code in Mandarin, Tamil, Jawi, Manglish, Johor, Kedah, Kelantan, Pahang, Perak, Sabah, Sarawak, Selangor, Negeri Sembilan and Terengganu.
Multi-turn Malaysian context such as related to Malaysian Legislation, politics, religions and languages.
Training session
Finetune on
mesolitica/Malaysian-SFT to make the model understand Malaysian context.
How we train
LoRA on ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "embed_tokens", "lm_head"].
Multipacking 8192 context length with proper SDPA causal masking to prevent document contamination and also make sure proper position ids.
Chunk CCE loss for LoRA.
Revision 83a0e145c726385502898ab7e016982eae1b684d
Rank 128, Alpha 256.
WanDB at https://wandb.ai/huseinzol05/lora-embedding-128-qwen2.5-7b-malaysian-8k
Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1
Rank 256, Alpha 512.
WanDB at https://wandb.ai/huseinzol05/lora-embedding-256-qwen2.5-7b-malaysian-8k
Benchmark
MalayMMLU
Probability next tokens
Based on 0-shot official MalayMMLU First token accuracy,
Revision 83a0e145c726385502898ab7e016982eae1b684d,
Model Accuracy shot by_letter category
0 Malaysian-Qwen2.5-7B-Instruct 72.042571 0shot True STEM
1 Malaysian-Qwen2.5-7B-Instruct 70.690204 0shot True Language
2 Malaysian-Qwen2.5-7B-Instruct 66.536571 0shot True Social science
3 Malaysian-Qwen2.5-7B-Instruct 67.306308 0shot True Others
4 Malaysian-Qwen2.5-7B-Instruct 71.808874 0shot True Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Malaysian-Qwen2.5-7B-Instruct
Metric : first
Shot : 0shot
average accuracy 69.26031470697559
accuracy for STEM 72.04257060990585
accuracy for Language 70.69020356234097
accuracy for Social science 66.53657126337092
accuracy for Others 67.30630846725833
accuracy for Humanities 71.80887372013652Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1,
While the original model,
Model Accuracy shot by_letter category
0 Qwen2.5-7B-Instruct 70.609906 0shot True STEM
1 Qwen2.5-7B-Instruct 68.034351 0shot True Language
2 Qwen2.5-7B-Instruct 63.486557 0shot True Social science
3 Qwen2.5-7B-Instruct 64.164068 0shot True Others
4 Qwen2.5-7B-Instruct 69.101251 0shot True Humanities
{'Social science': 6918, 'Language': 6288, 'Humanities': 4395, 'Others': 4169, 'STEM': 2443}
Model : Qwen2.5-7B-Instruct
Metric : first
Shot : 0shot
average accuracy 66.52211621856027
accuracy for STEM 70.60990585345887
accuracy for Language 68.03435114503816
accuracy for Social science 63.486556808326114
accuracy for Others 64.16406812185176
accuracy for Humanities 69.10125142207053First token match using vLLM
Based on 0-shot exact first token match using vLLM Guided Decoding,
Revision 83a0e145c726385502898ab7e016982eae1b684d,
Model Accuracy shot category
0 Malaysian-Qwen2.5-7B-Instruct 70.159640 0 STEM
1 Malaysian-Qwen2.5-7B-Instruct 66.682570 0 Language
2 Malaysian-Qwen2.5-7B-Instruct 62.893900 0 Social science
3 Malaysian-Qwen2.5-7B-Instruct 64.379947 0 Others
4 Malaysian-Qwen2.5-7B-Instruct 66.780432 0 Humanities
Model : Malaysian-Qwen2.5-7B-Instruct
Metric : full
Shot : 0
average accuracy 65.57221327386115
accuracy for STEM 70.15963978714696
accuracy for Language 66.68256997455471
accuracy for Social science 62.89389997108991
accuracy for Others 64.37994722955145
accuracy for Humanities 66.78043230944255Revision 5679143eadc2e7deb3bc61ec69ff301b2ba6a4e1,
While the original model,
Model Accuracy shot category
0 Qwen2.5-7B-Instruct 70.978305 0 STEM
1 Qwen2.5-7B-Instruct 68.177481 0 Language
2 Qwen2.5-7B-Instruct 64.238219 0 Social science
3 Qwen2.5-7B-Instruct 64.643799 0 Others
4 Qwen2.5-7B-Instruct 70.443686 0 Humanities
Model : Qwen2.5-7B-Instruct
Metric : full
Shot : 0
average accuracy 67.13748812621319
accuracy for STEM 70.97830536225952
accuracy for Language 68.17748091603053
accuracy for Social science 64.23821913847932
accuracy for Others 64.64379947229551
accuracy for Humanities 70.44368600682593Acknowledgement
Special thanks to
https://www.sns.com.my and Nvidia for 8x H100 node!
