Papers
arxiv:2407.09252

Context Embeddings for Efficient Answer Generation in RAG

Published on Jul 12
Authors:

Abstract

Retrieval-Augmented Generation (RAG) allows overcoming the limited knowledge of LLMs by extending the input with external information. As a consequence, the contextual inputs to the model become much longer which slows down decoding time directly translating to the time a user has to wait for an answer. We address this challenge by presenting COCOM, an effective context compression method, reducing long contexts to only a handful of Context Embeddings speeding up the generation time by a large margin. Our method allows for different compression rates trading off decoding time for answer quality. Compared to earlier methods, COCOM allows for handling multiple contexts more effectively, significantly reducing decoding time for long inputs. Our method demonstrates a speed-up of up to 5.69 times while achieving higher performance compared to existing efficient context compression methods.

Community

Paper author

We propose a compression method that reduces long contexts to only a handful of Context Embeddings speeding up the generation time by a large margin while maintaining high performance.

Sign up or log in to comment

Models citing this paper 4

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2407.09252 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 1