This is a MoE model project constructed based on the Qwen 1.8B model. In this project, we concatenated 4 original models and trained them using special training methods.

This model is a checkpoint model for the continue pretraining stage.

This project is open source under the Tongyi Qianwen Research License Agreement. You can view the complete license agreement in this link: LICENSE.

During the use of this project, please ensure that your usage behavior complies with the terms and conditions of the license agreement.