🍮 🦙 Flan-Alpaca: Instruction Tuning from Humans and Machines
Thanks to declare-lab for the training repository, contains code for extending the Stanford Alpaca
synthetic instruction tuning to existing instruction-tuned models such as Flan-T5.
The pretrained models and demos are available on HuggingFace 🤗 :
Alpaca represents an exciting new direction
to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily.
Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data.
The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model.
However, the original implementation is less accessible due to licensing constraints of the
underlying LLaMA model.
Furthermore, users have noted potential noise in the synthetic
dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but
less diverse) instructions such as Flan-T5.
Usage
from transformers import pipeline
prompt = "Write an email about an alpaca that likes flan"
model = pipeline(model="0-hero/flan-alpaca-ul2")
model(prompt, max_length=128, do_sample=True)
