Transformers have become the gold standard for many natural language processing tasks and, in particular, for multi-hop question answering (MHQA). This task includes processing a long document and reasoning over the multiple parts of it. The landscape of MHQA approaches can be classified into two primary categories. The first group focuses on extracting supporting evidence, thereby constraining the QA model's context to predicted facts. Conversely, the second group relies on the attention mechanism of the long input encoding model to facilitate multi-hop reasoning. However, attention-based token representations lack explicit global contextual information to connect reasoning steps. To address these issues, we propose GEMFormer, a two-stage method that first collects relevant information over the entire document to the memory and then combines it with local context to solve the task. Our experimental results show that fine-tuning a pre-trained model with memory-augmented input, including the most certain global elements, improves the model's performance on three MHQA datasets compared to the baseline. We also found that the global explicit memory contains information from supporting facts required for the correct answer.

翻译：Transformer已成为许多自然语言处理任务（特别是多跳问答）的金标准。该任务需处理长文档并对其多个部分进行推理。当前多跳问答方法主要分为两类：第一类聚焦于提取支撑证据，从而将问答模型的上下文限定在预测的事实内；第二类则依赖长输入编码模型的注意力机制来促进多跳推理。然而，基于注意力的词元表示缺乏显式的全局上下文信息来连接推理步骤。为解决这些问题，我们提出GEMFormer——一种两阶段方法，首先将整个文档中的相关信息收集至记忆模块，再将其与局部上下文结合来完成任务。实验结果表明，在基于记忆增强输入（包含最具确定性的全局元素）微调预训练模型后，模型在三个多跳问答数据集上的性能均优于基线。我们还发现，全局显式记忆包含正确答案所需的支撑事实信息。