caskcsg/

Llama-3-8B-LongMagpie-512K-Instruct

This repository contains the code, models and datasets for our paper [LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions].

High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.

Our released models are listed as follows. You can import these models by using HuggingFace's Transformers. All models are trained on long-context instruction data synthesized by fineweb-edu and Qwen/Qwen2.5-72B-Instruct model. In the following comparision, we choose Llama-3-8B-NExtLong-512K-Instruct as a baseline model, which is trained with Magpie instruction data. In addition, to maintain short-text performance, we propose a p-mix strategy that combines LongMagpie and UltraChat datasets, resulting in a performance-balanced model Llama-3-8B-LongMagpie-p-mix-512K-Instruct.

Our released datasets are listed as follows. All datasets are synthesized from the short-text datasets fineweb-edu.

Run the following script to install the remaining dependencies and train the model.

Refer to the HELMET, RULER, and Longbench V2 to evaluate the Instruct model.

Refer to infer_demo.py. Each line of data contains context extracted from fineweb-edu, query generated by LongMapgie and answer.

Based on LongMagpie_singledoc_longcontext_dataset, we used the MultiDoc method (refer to multidoc_format.py) to extend the context length and transformed it into SFT dialogue format.

Following ProLong, we concatenate the datasets to a fixed 64k context length and train using Document Mask technology.

We tokenized LongMagpie_multidoc_longcontext_dataset and concatenated it to a length of 64k (refer to build_sft_data.py), making it convenient to train using Document Mask technology. This dataset can be used to achieve the best long-text performance.

To balance these capabilities, we introduce \textit{p}-Mix, a novel instruction data hybridization strategy. The core idea is twofold. First, to emulate the typical non-contextual start of general tasks, we sample a short-context instruction at the beginning of each training sequence. Second, we append subsequent data segments probabilistically to construct a mixed-context sequence up to length $L_{max}$. With probability $P_L$, a long-context instruction (generated by LongMagpie) is chosen; otherwise, with probability $1-P_L$, another short-context sample is chosen. This process repeats until approaching the target sequence length, ensuring each instance starts with a short, context-free instruction followed by a dynamically mixed sequence of long and short segments.

If you have any questions related to the code or the paper, feel free to email Chaochen (gaochaochen@iie.ac.cn) and XingWu (wuxing@iie.ac.cn). If you encounter any problems when using the code, or want to report a bug, you can open an issue. Please try to specify the problem with details so we can help you better and quicker!