Retrieval-based augmentations (RA) incorporating knowledge from an external database into language models have greatly succeeded in various knowledge-intensive (KI) tasks. However, integrating retrievals in non-knowledge-intensive (NKI) tasks is still challenging. Existing works focus on concatenating retrievals with inputs to improve model performance. Unfortunately, the use of retrieval concatenation-based augmentations causes an increase in the input length, substantially raising the computational demands of attention mechanisms. This paper proposes a new paradigm of RA named \textbf{ReFusion}, a computation-efficient Retrieval representation Fusion with bi-level optimization. Unlike previous works, ReFusion directly fuses the retrieval representations into the hidden states of models. Specifically, ReFusion leverages an adaptive retrieval integrator to seek the optimal combination of the proposed ranking schemes across different model layers. Experimental results demonstrate that the proposed ReFusion can achieve superior and robust performance in various NKI tasks.

翻译：检索增强方法通过将外部数据库的知识融入语言模型，在众多知识密集型任务中取得了显著成功。然而，在非知识密集型任务中集成检索机制仍面临挑战。现有研究主要关注将检索结果与输入进行拼接以提升模型性能。遗憾的是，这种基于检索拼接的增强方式会导致输入长度增加，从而显著提高注意力机制的计算开销。本文提出了一种名为\textbf{ReFusion}的新型检索增强范式，即采用双层优化的计算高效检索表示融合方法。与先前工作不同，ReFusion直接将检索表示融合至模型的隐藏状态中。具体而言，该方法通过自适应检索集成器，在不同模型层级间寻找所提出的排序方案的最优组合。实验结果表明，所提出的ReFusion能够在多种非知识密集型任务中实现优异且稳健的性能。