[2024/03/25] Released the XVERSE-65B-Chat GGUF models, supporting llama.cpp to inference the XVERSE-65B-Chat model on MacOS/Linux/Windows systems.
[2023/12/08] Released the XVERSE-65B-2 base model. This model builds upon its predecessor through Continual Pre-Training, reaching a total training volume of 3.2 trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a 20% improvement on the GSM8K benchmark and a 41% increase on HumanEval.
[2023/11/29] Update model architecture and additional pre-training data information.
[2023/11/24] Update the related information of the pre-training data.
[2023/11/06] Released the XVERSE-65B base model.
模型介绍
XVERSE-65B 是由深圳元象科技自主研发的支持多语言的大语言模型（Large Language Model），参数规模为 650 亿，本次开源的模型为底座模型 XVERSE-65B，主要特点如下：
XVERSE-65B is a multilingual large language model, independently developed by Shenzhen Yuanxiang Technology. The models released this time is the base model XVERSE-65B. Its key features are as follows:
Model Structure: XVERSE-65B uses the mainstream Decoder-only Transformer network structure, supports 16k context length, which can meet the need of longer multi-round dialogues, knowledge question-answering, and summarization. This makes the model more versatile in application scenarios.
Training Data: The model has been thoroughly trained on a diversified and high-quality dataset consisting of 2.6 trillion of tokens, including more than 40 languages such as Chinese, English, Russian, and Spanish. The sampling ratio of different types of data is finely set, which makes the performance of Chinese and English excellent, and also takes into account the effect of other languages.
Tokenization: Based on the BPE (Byte-Pair Encoding) algorithm, a tokenizer with a vocabulary size of 100,534 has been trained using hundreds of gigabytes of language data. This tokenizer is capable of supporting multilingual without the need for additional vocabulary expansion.
Training Framework: The training utilizes FlashAttention2 for accelerated computation, and on top of 3D parallelism, virtual pipeline technology is applied to reduce the excessive bubble rate caused by longer pipelines and 16k context windows. This achieves a peak computational efficiency within the industry-leading range in the petaflop-scale cluster. Concurrently, through continuous optimization of cluster infrastructure operations, resource scheduling, training frameworks, and the scheduling platform, a highly stable, low-interruption, and robust fault-tolerant training system has been developed, enhancing the effective weekly training rate to 98.6%.
The models sizes, architectures and learning rate of XVERSE-65B are showed as follows:
Cloning the repo may be inefficient, and thus you can manually download the GGUF file that you need or use huggingface-cli (pip install huggingface_hub) as shown below:
For the q5_k_m model, due to maximum file size for uploading, we split the GGUF file into 2. Essentially, we split a byte string to 2, and thus you can just concatenate them to get the whole file:
Like all other Large Language Models (LLMs), XVERSE-65B may produce inaccurate, biased, or otherwise offensive content under certain circumstances. Therefore, please use the content generated by the model with caution and refrain from disseminating harmful content. Before deploying any application of XVERSE-65B, developers should conduct safety tests and optimization of the model according to its specific application.
We strongly warn against the use of the XVERSE-65B model for producing or spreading harmful information, or conducting any activities that might harm the public, national, or social security, or violate regulations. We assume no responsibility for any problems arising from the use of the XVERSE-65B model, whether it be data security issues, public opinion risks, or any risks and issues caused by misunderstanding, misuse, dissemination, or non-compliance with the model.
The use of the source code in this repository must follow the Apache-2.0 open-source license, while the use of the model weights of XVERSE-65B needs to adhere to the Model License Agreement.
The XVERSE-65B model weights are fully open to academic research and support free commercial use. To apply for a commercial license, please fill in the application form. For other questions or collaborations, please contact opensource@xverse.cn.
