FullAttention-2.7b-4k is a model fine-tuned from facebook/opt-2.7b following the context window extension method described in Adapting Language Models to Compress Contexts. The 2,048 positional embeddings of the pre-trained OPT-2.7b are duplicated and the model is fine-tuned on sequences of 4,096 tokens from 2B tokens from The Pile.

To get started, download the AutoCompressor repository and load the model as follows:

from auto_compressor import AutoCompressorModel

model = AutoCompressorModel.from_pretrained("princeton-nlp/FullAttention-2.7b-4k")

Evaluation

We record the perplexity achieved by our OPT-2.7b models on segments of 2,048 tokens, conditioned on different amounts of context. FullAttention-2.7-4k uses full uncompressed contexts whereas AutoCompressor-2.7b-6k and RMT-2.7b-8k compress segments of 2,048 tokens into 50 summary vectors.

In-domain Evaluation

Context Tokens	0	512	2048	4096	6144
FullAttention-2.7b-4k	6.57	6.15	5.94	-	-
RMT-2.7b-8k	6.34	6.19	6.02	6.02	6.01
AutoCompressor-2.7b-6k	6.31	6.04	5.98	5.94	5.93

Out-of-domain Evaluation

Context Tokens	0	512	2048	4096	6144
FullAttention-2.7b-4k	8.94	8.28	7.93	-	-
RMT-2.7b-8k	8.62	8.44	8.21	8.20	8.20
AutoCompressor-2.7b-6k	8.60	8.26	8.17	8.12	8.10

Bibtex

@misc{chevalier2023adapting,
      title={Adapting Language Models to Compress Contexts}, 
      author={Alexis Chevalier and Alexander Wettig and Anirudh Ajith and Danqi Chen},
      year={2023},
      eprint={2305.14788},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}