The Instella-Long dataset is a collection of pre-training and instruction following data that is used to train Instella-3B-Long-Instruct. The pre-training data is sourced from Prolong. For the SFT data, we use public datasets: Ultrachat 200K, OpenMathinstruct-2, Tülu-3 Instruction Following, and MMLU auxiliary train set. In addition, we generate synthetic long instruction data using documents of the books and arxiv from our pre-training corpus and the dclm subset from… See the full description on the dataset page:
https://huggingface.co/datasets/amd/Instella-Long.
