Previous work finds that recent long-context language models fail to make equal use of information in the middle of their inputs, preferring pieces of information located at the tail ends which creates an undue bias in situations where we would like models to be equally capable of using different parts of the input. Thus far, the problem has mainly only been considered in settings with single pieces of critical information, leading us to question what happens when multiple necessary pieces of information are spread out over the inputs. Here, we demonstrate the effects of the "lost in the middle" problem in the multi-hop question answering setting -- in which multiple reasoning "hops" over disconnected documents are required -- and show that performance degrades not only with respect to the distance of information from the edges of the context, but also between pieces of information. Additionally, we experiment with means of alleviating the problem by reducing superfluous document contents through knowledge graph triple extraction and summarization, and prompting models to reason more thoroughly using chain-of-thought prompting.

翻译：先前的研究发现，近期出现的具有长上下文处理能力的语言模型无法均等地利用输入信息中间部分的内容，而是倾向于使用位于输入首尾两端的信息片段，这在需要模型同等利用输入不同部分的场景中产生了不当的偏差。迄今为止，该问题主要仅在包含单一关键信息片段的情境中被探讨，这促使我们思考：当多个必要信息片段分散在输入内容中时会发生什么情况。本文通过多跳问答场景——该场景要求对互不关联的文档进行多次推理"跳跃"——揭示了"迷失于中间"问题的具体影响，并证明性能下降不仅与信息距上下文边界的距离相关，还与信息片段之间的间隔距离有关。此外，我们尝试通过知识图谱三元组抽取与摘要技术来缩减冗余文档内容，并采用思维链提示方法引导模型进行更深入的推理，从而探索缓解该问题的可行途径。