This is a Quark-based baseline developed during the research carried out in the ALT paper. The model is trained following the algorithm introduced in Quark, with a slight modification as to sample multiple generations per prompt to compute the reward quantiles locally instead of globally across all prompts. We found that this was crucial for training. Notice that Quark was not introduced for tackling the alignment problem but for unlearning attributes in text completion tasks, such as unlearning toxcity, negative sentiment or repetition.

It is a fine-tuned GPT-J (6B) model on the TL;DR Summarization dataset to be better aligned with humans' preferences on summaries, i.e., accounting for axes such as accuracy, coverage, and coherence.

The alignment process departs from a SFT checkpoint released by CarperAI and trained using their trlx library.

In a nutshell, the Quark method consists on sampling new generations and scoring them with a reward model to further cluster them into reward quantiles. For every quantile in a pre-defined number of quantiles, a new reward quantile token is added to the tokenizer. Afterward, each generation is mapped to a reward quantile token, and the latter is preppended to the input prompt for conditional language modelling training.

For extensive coverage on Quark, please refer to their paper.

The reward model used for scoring the generations can be found in here. We used K = 5 quantile tokens, which were newly added to the tokenizer:

Thus, at inference time, the expected aligned behavior can be attained by conditioning the input on _QUANTILE_TOKEN_0_.

Related Models: ALT-RM.

This model originates from a research project focused on alignment and is intended primarily for research purposes. Commercial use as an off-the-shelf model is discouraged, as it was not designed with such applications in mind. The model is tailored specifically for the summarization task, having been trained on the TL;DR dataset, though some out-of-distribution generalization may be possible for related datasets.

You should format the input by preppending the feedback as follows: _QUANTILE_TOKEN_0_{prompt}

The model was trained on the TL;DR summarization dataset introduced in the Stiennon et al.'s, "Learning to Summarize from human feedback" paper. We employed the dataset version from CarperAI, which can be found in the HuggingFace Hub in here.

The exact training procedure and hyper-parameters configuration can be found in our paper.

As an evaluation metric, we compute GPT-4 win-rates over PPO on a 1k random subset of the test set. We use the prompt provided in the DPO paper and we ask GPT-4 to compare generations between ALT-RM and Quark and PPO. Furthermore, we report the following metrics computed on the whole test set: average reward model score, perplexity measured by the SFT reference policy as a proxy for fluency, and average length of the generations. In addition, we conduct an out-of-domain evaluation and compute GPT-4 win-rates on 100 articles from the test split of the CNN/DailyMail dataset.

@misc{lloret2024aligninglanguagemodelstextual,
      title={Towards Aligning Language Models with Textual Feedback}, 
      author={Saüc Abadal Lloret and Shehzaad Dhuliawala and Keerthiram Murugesan and Mrinmaya Sachan},
      year={2024},
      eprint={2407.16970},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.16970}, 
}