Our data preprocessing and pre-training pipeline is designed for enhanced training efficiency and model quality,
achieved through several key innovations:
-
Dataset Quality Benchmarking: A quantile benchmarking approach applied to major open-source pretraining datasets (e.g., DCLM Baseline, Fineweb-Edu)
reveals their quality distributions via small-scale training runs, informing better data selection.
-
Multi-Phase Pre-Training: The training progresses through 5 phases, strategically increasing the ratio of reasoning-intensive and knowledge-intensive samples
while selectively repeating high-quality data portions.
-
Multi-Domain Curriculum Learning: We keep a stable data mixture across different datasets while ordering samples within each dataset by ascending quality.
This curriculum is further leveraged through
accommodated learning rate decay and model averaging.
-
High-Performance Data Preprocessing: We built an open-source,
Spark-based framework optimized with
Chukonu,
delivering exceptional efficiency for large-scale deduplication and sorting tasks.
-
Architecture for Training Stability: Optimized for training on Ascend 910A clusters (FP16 precision, similar to V100),
the Kaiyuan-2B architecture integrates QK norm, sandwich norm, and soft-capping techniques to ensure stable and robust pre-training.
The model architecture is similar to
Qwen/Qwen3-1.7B,
and can be easily loaded by libraries like
transformers.
Please cite
our technical report if you use our model, dataset, or code.
1@misc{luo2025pcmind21kaiyuan2btechnicalreport,
2 title={PCMind-2.1-Kaiyuan-2B Technical Report},
3 author={Kairong Luo and Zhenbo Sun and Xinyu Shi and Shengqi Chen and Bowen Yu and Yunyi Chen and Chenyi Dang and Hengtao Tao and Hui Wang and Fangming Liu and Kaifeng Lyu and Wenguang Chen},
4 year={2025},
5 eprint={2512.07612},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2512.07612},
9}
All artifacts (including code, model weights, and training data) of Kaiyuan-2B
are licensed under
Apache-2.0 License with the following copyright notice:
1Copyright 2025 Tsinghua University & Peng Cheng Laboratory
2
3Licensed under the Apache License, Version 2.0 (the "License");
4you may not use this file except in compliance with the License.
5You may obtain a copy of the License at
6
7 http://www.apache.org/licenses/LICENSE-2.0
8
9Unless required by applicable law or agreed to in writing, software
10distributed under the License is distributed on an "AS IS" BASIS,
11WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12See the License for the specific language governing permissions and
13limitations under the License.
