Pre-trained language model with identical parameters to gpt-neo-125m, but with additional language modeling heads ("exits") connected to different layers of the model.

These 6 additional heads (in layers 2, 4, 6, 8, 10, 12) were trained on the English portion of CC-100 while keeping the original pre-trained model parameters frozen.

The model can be used for the Autocontrastive Decoding text generation approach described in Gera et al. 2023, for early-exiting approaches, or for other algorithms that consider the next-token predictions of different model layers.

Harnessing the additional language modeling heads requires loading the model using the auto-contrastive-generation library (pip install autocontrastive-gen).

In a nutshell, the user creates a MultiExitConfiguration that determines model behavior at training and inference, and then loads the model using the dedicated AutoMultiExitModel class. After that, the model can be used with the transformers API like any other model. See the GitHub for detailed usage instructions.

For example, the code below initializes the model to use Autocontrastive Decoding, and then performs text generation in this chosen setting:

Ariel Gera, Roni Friedman, Ofir Arviv, Chulaka Gunasekara, Benjamin Sznajder, Noam Slonim and Eyal Shnarch. The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers. ACL 2023.