Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low. Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors. In this paper, we study the state-of-the-art Retro model and observe that its performance gain is better explained by surface-level similarities, such as token overlap. Inspired by this, we replace the semantic retrieval in Retro with a surface-level method based on BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.

翻译：研究表明，通过检索机制增强语言模型可以在保持参数数量较少的同时显著提升其性能。检索增强模型通常依赖于基于查询块与潜在相邻块的稠密表示间相似性的语义检索机制。本文研究了当前最先进的Retro模型，发现其性能提升更合理地归因于表面层面的相似性（如词元重叠）。受此启发，我们将Retro模型中的语义检索替换为基于BM25的表面检索方法，显著降低了困惑度。由于针对大规模数据集进行完整BM25检索计算成本较高，我们将其应用于重排序场景，以极小的计算开销获得了部分困惑度降低的效果。