The EntiGraph CPT model is a continuation of the Llama 3 8B base model, trained using the Synthetic Continued Pretraining by Yang et al. (2024) approach with the EntiGraph algorithm. This model has been trained on a synthetic corpus generated from the QuALITY dataset to acquire domain-specific knowledge efficiently.
The code used to train the model is available at the Synthetic Continued Pretraining GitHub repo.
Model Details
Developed by: Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto
Model type: Causal Language Model
Language(s): English
License: Apache 2.0
Finetuned from model: Llama 3 8B
Uses
Intended Use
This model is intended for research purposes and applications requiring domain-specific knowledge related to the QuALITY dataset. It can be used for tasks such as closed-book question answering, summarization, and other NLP tasks within the domain of the training data.
Out-of-Scope Use
This model should not be used for generating factual information outside the scope of its training data or for any malicious purposes.
Training Details
Training Data
The model was trained on a 455M token synthetic corpus generated by the EntiGraph algorithm from the QuALITY dataset.
Training Procedure
Pretraining: Continued pretraining on the EntiGraph synthetic corpus
Hyperparameters:
Learning rate: 5e-06
Batch size: 16
Weight decay: 0.01
Warmup: 0.05
Epochs: 2
RedPajama replay rate: 0.1
Evaluation
The model has been evaluated on the QuALITY question answering dataset, demonstrating improved performance in closed-book QA tasks compared to the base model.
Limitations and Biases
While the EntiGraph CPT model shows improved performance on domain-specific tasks, it may inherit biases present in the original Llama 3 8B model and the QuALITY dataset. Users should be aware of potential limitations in generating content outside its training domain.
Citation
If you use this model, please cite the original paper:
@misc{yang2024syntheticcontinuedpretraining,
title={Synthetic continued pretraining},
author={Zitong Yang and Neil Band and Shuangping Li and Emmanuel Candès and Tatsunori Hashimoto},
year={2024},
eprint={2409.07431},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2409.07431},
}
Ethical Considerations
Users of this model should be aware of the ethical implications of using large language models and ensure responsible use in applications.
