In LongLoRA approach, The proposed shifted short attention is easy to implement, compatible with Flash-Attention, and is not required during inference.
We built up a long-context instruction-following dataset, LongAlpaca-12k. We released the corresponding LongAlpaca-7B, LongAlpaca-13B and LongAlpaca-70B models. To our best knowledge, this is the first open-sourced long-context 70B model.
LongAlpaca-12k contains 9k long QA data that we collected and 3k short QA sampled from the original Alpaca data. This is to avoid the case that the model might degrade at short instruction following. The data we collect contains various types and amounts as the following figure.
Following the original Alpaca format, our Long QA data uses the following prompts for fine-tuning:
instruction: str, describes the task the model should perform. For example, to answer a question after reading a book section or paper. We vary the contents and questions to make instructions diverse.
output: str, the answer to the instruction.
We did not use the input format in the Alpaca format for simplicity.
This project also supports GPTNeoX models as the base model architecture. Some candidate pre-trained weights may include GPT-NeoX-20B, Polyglot-ko-12.8B and other variants.
Please remember to change path_to/Llama-2-7b-hf, path_to_saving_checkpoints, path_to_cache to your own directory.
Note that you can change model_max_length to other values.
You could change ds_configs/stage2.json to ds_configs/stage3.json if you want.
Please set use_flash_attn as False if you use V100 machines or do not install flash attention.
You can set low_rank_training as False if you want to use fully fine-tuning. It will cost more GPU memory and slower, but the performance will be a bit better.
When training is finished, to get the full model weight:
cd path_to_saving_checkpoints && python zero_to_fp32.py . pytorch_model.bin
There is no need to make supervised fine-tuning upon the fine-tuned context extended models. It is all right to directly use base model as Llama2-chat models, as the amount of long instruction following data is enough for SFT.
In low-rank training, we set embedding and normalization layers as trainable. Please use the following line to extract the trainable weights trainable_params.bin from pytorch_model.bin
Merge the LoRA weights of pytorch_model.bin and trainable parameters trainable_params.bin, save the resulting model into your desired path in the Hugging Face format:
To evaluate a model that is trained in the low-rank setting, please set both base_model and peft_model. base_model is the pre-trained weight. peft_model is the path to the saved checkpoint, which should contain trainable_params.bin, adapter_model.bin and adapter_config.json. For example,
To evaluate a model that is fully fine-tuned, you only need to set base_model as the path to the saved checkpoint, which should contain pytorch_model.bin and config.json. peft_model should be ignored.
Note that --seq_len is to set the sequence length for evaluation. --context_size is to set the context length of the model during fine-tuning. --seq_len should not be larger than --context_size.
We have already tokenized the validation and test splits of PG19 and proof-pile dataset into pg19/validation.bin, pg19/test.bin, and proof-pile/test_sampled_data.bin, with the tokenizer of LLaMA. proof-pile/test_sampled_data.bin contains 128 documents that are randomly sampled from the total proof-pile test split. For each document, it has at least 32768 tokens. We also release the sampled ids in proof-pile/test_sampled_ids.bin. You can download them from the links below.
python3 inference.py \
--base_model /data/models/Llama-2-13b-chat-longlora-32k-sft \
--question "Why doesn't Professor Snape seem to like Harry?" \
--context_size 32768 \
--max_gen_len 512 \
--flash_attn True \
--material "materials/Harry Potter and the Philosophers Stone_section2.txt" \
--material_type "book" \
--material_title "Harry Potter and the Philosophers Stone"
Note that you can ignore material_type or material_title.
To ask a question related to a paper:
python3 inference.py \
--base_model /data/models/Llama-2-13b-chat-longlora-32k-sft \
--question "What are the main contributions and novelties of this work?" \
--context_size 32768 \
--max_gen_len 512 \
--flash_attn True \
--material "materials/paper1.txt" \
--material_type "paper"
Note that flash_attn=True will make the generation slow but save much GPU memory.
Data Generation via Pdf2text
During our dataset collection, we convert paper and books from pdf to text. The conversion quality has a large influence on the final model quality. We think that this step is non-trivial. We release the tool for the pdf2txt conversion, in the folder pdf2txt. It is built upon pdf2image, easyocr, ditod and detectron2. Please refer to the README.md in pdf2txt for more details.
Citation
If you find this project useful in your research, please consider citing:
@article{longlora,
title={LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models},
author={Yukang Chen and Shengju Qian and Haotian Tang and Xin Lai and Zhijian Liu and Song Han and Jiaya Jia},
journal={arXiv:2309.12307},
year={2023}
}
@misc{long-alpaca,
author = {Yukang Chen and Shaozuo Yu and Shengju Qian and Haotian Tang and Xin Lai and Zhijian Liu and Song Han and Jiaya Jia},
title = {Long Alpaca: Long-context Instruction-following models},
year = {2023},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/dvlab-research/LongLoRA}},
}
Acknowledgement
This work is built upon the LLaMA2 as the pre-trained models.
This work can also be built upon the GPTNeoX-HF which is based upon EleutherAI/GPTNeoX as the pre-trained model architecture.
LongLoRA is licensed under the Apache License 2.0. This means that it requires the preservation of copyright and license notices.
Data and weights are under CC-BY-NC 4.0 License. They are licensed for research use only, and allowed only non-commercial. Models trained using the dataset should not be used outside of research purposes.
