YuYan is a series of natural language processing models developed by Fuxi AI lab, Netease.Inc, including text generation models, natural language understanding models, and more. YuYan-10b is a natural language understanding model trained on high-quality Chinese corpus.
YuYan-10b is similar to BERT in that it is trained on large-scale pre-training corpora using unsupervised learning. However, it differs in that it incorporates various tasks such as sentence order and word deletion in addition to the MLM task during training to enhance the model's semantic representation ability and improve its understanding of Chinese.
CLUE result
Score
AFQMC
TNEWS1.1
IFLYTEK
OCNLI_50k
WSC1.1
CSL
YuYan-10b
84.685
86.45
74.04
67.96
86.33
95.73
97.6
HUMAN
84.1
81
71
80.3
90.3
98
84
HunYuan-NLP 1T
83.632
85.11
70.44
67.54
86.5
96
96.2
How to use
Our model is trained based on the Megatron. As a result, the inference and finetuning depend on it.
Below are the install tutorial.
We have packaged all the required dependencies for the model. Use the following command to obtain the model running environment.
pip install -r requirements.txt
Finetuning script
We provide multiple scripts for finetuning on the CLUE benchmark, which is a Chinese language understanding evaluation leaderboard that covers various tasks such as natural language understanding, reading comprehension, and semantic understanding. For any given CLUE task, use the following command to start finetuning.
# finetuning afqmc task
sh finetune_afqmc_distributed.sh
# finetuning csl task
sh finetune_csl_distributed.sh
