CPM (Chinese Pre-trained Language Model) is a Transformer-based autoregressive language model, with 2.6 billion parameters and 100GB Chinese training data. To the best of our knowledge, CPM is the largest Chinese pre-trained language model, which could facilitate downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. [Project] [Model] [Paper]

The text generated by CPM is automatically generated by a neural network model trained on a large number of texts, which does not represent the authors' or their institutes' official attitudes and preferences. The text generated by CPM is only used for technical and scientific purposes. If it infringes on your rights and interests or violates social morality, please do not propagate it, but contact the authors and the authors will deal with it promptly.

We collect different kinds of texts in our pre-training, including encyclopedia, news, novels, and Q&A. The details of our training data are shown as follows.

Based on the hyper-parameter searching on the learning rate and batch size, we set the learning rate as \(1.5\times10^{-4}\) and the batch size as \(3,072\), which makes the model training more stable. In the first version, we still adopt the dense attention and the max sequence length is \(1,024\). We will implement sparse attention in the future. We pre-train our model for \(20,000\) steps, and the first \(5,000\) steps are for warm-up. The optimizer is Adam. It takes two weeks to train our largest model using \(64\) NVIDIA V100.

We evaluate CPM with different numbers of parameters (the details are shown above) on various Chinese NLP tasks in the few-shot (even zero-shot) settings. With the increase of parameters, CPM performs better on most datasets, indicating that larger models are more proficient at language generation and language understanding. We provide results of text classification, chinese idiom cloze test, and short text conversation generation as follows. Please refer to our paper for more detailed results.