COCOM is an effective context compression method, reducing long contexts to only a handful of Context Embeddings speeding up the generation time for Question Answering

Retrieval-Augmented Generation (RAG) allows overcoming the limited knowledge of LLMs by extending the input with external context. A major drawback in RAG is the considerable increase in decoding time with longer inputs. We address this challenge by presenting COCOM, an effective context compression method, reducing long contexts to only a handful of Context Embeddings speeding up the generation time. Our method allows for different compression rates trading off decoding time for answer quality. Compared to earlier methods, COCOM allows for handling multiple contexts more effectively, significantly reducing decoding time for long inputs. Our method demonstrates a speed-up of up to 5.69 times while achieving higher performance compared to existing efficient context compression methods.

For batch processing, the model takes as input

The model compresses the questions into context embeddings and answers the question based on the provided context embeddings.

Paper: https://arxiv.org/pdf/2407.09252

@misc{rau2024contextembeddingsefficientanswer,
      title={Context Embeddings for Efficient Answer Generation in RAG}, 
      author={David Rau and Shuai Wang and Hervé Déjean and Stéphane Clinchant},
      year={2024},
      eprint={2407.09252},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.09252}, 
}