This model follows the Humpback architecture, proposed in the paper
Self-Alignment with Instruction Backtranslation
by Li et al.
It represents the resulting model after the first iteration of self-curation, which is trained on a small amount of gold data
and a set of generated data curated by the
"seed model".
This model can be used for instruction-following.
It may also be used to, again, score the instruction-response pairs
generated by the
"backward model" for a second iteration of self-curation.
Humpback uses instruction backtranslation on a web corpus to generate input-output pairs (self-augmentation),
creating a richer dataset for fine-tuning models without the need for additional manual annotation.
The model then iteratively curates the created dataset, scoring the pairs by quality, and is then finetuned on the resulting subset
of all pairs with the highest possible score (self-curation).
Varying from the original paper, this model is a fine-tuned version of
meta-llama/Llama-3.2-3B.
It has been trained using
TRL.
The dataset used to train this model is a combination of data sampled from the
oasst1
dataset and the synthetic dataset which was mentioned above. The latter has been created by applying self-augmentation and self-curation
on 502k entries from the english subset ("en") of the
c4 dataset.
For comparison with other methods, the training dataset was limited to 16000 instruction-response pairs.
1@misc{li2023selfalignment,
2 title={Self-Alignment with Instruction Backtranslation},
3 author={Xian Li and Ping Yu and Chunting Zhou and Timo Schick and Luke Zettlemoyer and Omer Levy and Jason Weston and Mike Lewis},
4 year={2023},
5 eprint={2308.06259},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
1@misc{vonwerra2022trl,
2 title = {{TRL: Transformer Reinforcement Learning}},
3 author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
4 year = 2020,
5 journal = {GitHub repository},
6 publisher = {GitHub},
7 howpublished = {\url{https://github.com/huggingface/trl}}
8}
