[Paper] [HF Collection] [Code]
ProLong (Princeton long-context language models) is a family of long-context models that are continued trained and supervised fine-tuned from Llama-3-8B, with a maximum context window of 512K tokens. Our main ProLong model is one of the best-performing long-context models at the 10B scale (evaluated by HELMET).
To train this strong long-context model, we conduct thorough ablations on the long-context pre-training data… See the full description on the dataset page:
https://huggingface.co/datasets/princeton-nlp/prolong-data-64K.
