GALACTICA 6.7B fine-tuned on the Alpaca dataset.

The model card from the original Galactica repo can be found here, and the original paper here.

The dataset card for Alpaca can be found here, and the project homepage here. The Alpaca dataset was collected with a modified version of the Self-Instruct Framework, and was built using OpenAI's text-davinci-003 model. As such it is subject to OpenAI's terms of service.

The GALACTICA models are trained on a large-scale scientific corpus and are designed to perform scientific tasks. The Alpaca dataset is a set of 52k instruct-response pairs designed to enhace the instruction following capabilites of pre-trained language models.

The GALACTICA model card specifies that the primary indended users of the GALACTICA models are researchers studying language models applied to the scientific domain, and it cautions against production use of GALACTICA without safeguards due to the potential for the model to produce inaccurate information. The original GALACTICA models are available under a non-commercial CC BY-NC 4.0 license, and the GALPACA model is additionally subject to the OpenAI Terms of Service.

The GALPACA models are trained by fine-tuning pre-trained GALACTICA models on the Alpaca dataset. GALACTICA models were trained on 106 billion tokens of open-access scientific text and data, including papers, textbooks, scientific websites, encyclopedias, and more. Fine-tuning the base GALACTICA models on the 52k instruction-response pairs in the Alpaca dataset allows users to query the GALPACA models in an instruct-response fashion.

The GALPACA weights are made available for use with the transformers library.

GALPACA 6.7B was fine-tuned in about 2 hours using 4 A100 80GB GPUS, 16-bit mixed-precision, an effective batch-size of 128, and with a maximum context window of 512 tokens. This model was trained using full-shard data parallelism.

Qualitative evaluation suggests that Galpaca frequently outperforms LLaMA-based Alpaca models on tasks related to technical knowledge and programming, while it underperforms on natural langauge tasks such as generating prose. More information about the performance and limitations of the GALACTICA family of models can be found on the original GALACTICA model card.