We explore supervised multitask pre-training by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train language models. The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models. Instruction Pre-Training outperforms Vanilla Pre-training in both general pre-training from scratch and domain-adaptive continual pre-training. In pre-training from scratch, Instruction Pre-Training not only improves pre-trained base models but also benefits more from further instruction tuning. In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B.

**************************** Updates ****************************

2026/1/23: Released LLM-in-Sandbox Elicits General Agentic Intelligence, where the data of Instruction Pre-Training achieves robust generalization in agentic RL!
2024/11/30: Released the multimodal version of the instruction synthesizer: Visual Instruction Synthesizer
2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉
2024/9/11: Updated FAQ on continual pre-training from Llama3
2024/8/29: Updated guidelines on evaluating any 🤗Huggingface models on the domain-specific tasks
2024/7/31: Updated pre-training suggestions in the Advanced Usage section of instruction-synthesizer
2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:

* 2024/6/21: Released the [paper](https://huggingface.co/papers/2406.14491), [code](https://github.com/microsoft/LMOps), and [resources](https://huggingface.co/instruction-pretrain)

Resources

🤗 We share our data and models with example usages, feel free to open any discussions at this page! 🤗

Thanks to the demo davanstrien/instruction-synthesizer for implementing our approach
Context-Based Instruction Synthesizer: instruction-synthesizer
Fine-Tuning Data for the Synthesizer: ft-instruction-synthesizer-collection
General Models Pre-Trained from Scratch (on 100B tokes):
- InstructLM-500M
- InstructLM-1.3B
Domain-Specific Models Pre-Trained from Llama3-8B:
- Finance-Llama3-8B
- Biomedicine-Llama3-8B
General Instruction-Augmented Corpora: general-instruction-augmented-corpora
Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): medicine-instruction-augmented-corpora

Domain-Adaptive Continued Pre-Training

Following AdaptLLM, we augment the domain-specific raw corpora with instruction-response pairs generated by our context-based instruction synthesizer.

1. To chat with the biomedicine-Llama3-8B model:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/medicine-Llama3-8B")
tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/medicine-Llama3-8B")

# Put your input here, NO prompt template is required
user_input = '''Question: Which of the following is an example of monosomy?
Options:
- 46,XX
- 47,XXX
- 69,XYY
- 45,X

Please provide your choice first and then provide explanations if possible.'''

inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]

answer_start = int(inputs.shape[-1])
pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)

print(pred)

2. To evaluate any Huggingface LMs on domain-specific tasks (💡New!)

You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).

1). Set Up Dependencies

bash

git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt

2). Evaluate the Model

bash

# Select the domain from ['biomedicine', 'finance']
DOMAIN='biomedicine'

# Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates)
MODEL='instruction-pretrain/medicine-Llama3-8B'

# Model parallelization:
# - Set MODEL_PARALLEL=False if the model fits on a single GPU. 
#   We observe that LMs smaller than 10B always meet this requirement.
# - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU.
MODEL_PARALLEL=False

# Choose the number of GPUs from [1, 2, 4, 8]
N_GPU=1

# Whether to add a BOS token at the beginning of the prompt input:
# - Set to False for AdaptLLM.
# - Set to True for instruction-pretrain models.
# If unsure, we recommend setting it to False, as this is suitable for most LMs.
add_bos_token=True

# Run the evaluation script
bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}

FAQ on Continual Pre-Training from LLama3

Q1: Do you use the official Llama3 instruction prompt for pre-training?

No, the provided Llama3 instruction prompt is designed for the instruction-tuned model, but our continual pre-training is conducted on the pre-trained base model where only BOS (<|begin_of_text|>) and EOS (<|end_of_text|>) tokens are required.

Q2: For the general instructions from OpenOrca, do you concatenate each instruction with its output using '\n'?

No, as mentioned in the pre-training suggestions, we use a simple whitespace to concatenate each question with its response for the general instruction data from OpenOrca. This is because OpenOrca's data is already templated with diverse natural languge templates (such as those with \n), so a whitespace is sufficient to formulate the data.

Note that when using our templated instruction-augmented texts, you don't need to add any concatenations.

Q3: What about those system prompts in OpenOrca?

We simply discard the system prompts.

To put it all together, the text before tokenization looks like this:

python

general_instruction_response_text = "<|begin_of_text|>{question} {response}<|end_of_text|>"

instruction_augmented_text = "<|begin_of_text|>{instruction augmented text}<|end_of_text|>"

Then, for tokenization, you don't need to add BOS and EOS token ids. The tokenization code looks like this:

text_ids = tokenizer(text, add_special_tokens=False, **kwargs).input_ids

Citation

If you find our work helpful, please cite us:

Instruction Pre-Training (EMNLP 2024)

bibtex

@article{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
  journal={arXiv preprint arXiv:2406.14491},
  year={2024}
}

Adapt LLM to Domains（ICLR 2024）

bibtex

@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}