Making the contents generated by Large Language Model (LLM) such as ChatGPT, accurate, credible and traceable is crucial, especially in complex knowledge-intensive tasks that require multi-step reasoning and each of which needs knowledge to solve. Introducing Information Retrieval (IR) to provide LLM with external knowledge is good potential to solve this problem. However, where and how to introduce IR into LLM is a big challenge. Previous work has the disadvantage that the wrong knowledge retrieved by IR misleads the LLM or breaks the reasoning chain of LLM. In this paper, we propose a novel framework called Search-in-the-Chain (SearChain) for the interaction between LLM and IR to solve the challenges. First, LLM generates the global reasoning chain called Chain-of-Query (CoQ) where each node consists of an IR-oriented query and the answer to the query. Second, IR verifies the answer of each node of CoQ, it corrects the answer that is not consistent with the retrieved information when IR gives high confidence, which improves the credibility. Third, LLM can mark its missing knowledge in CoQ and IR can provide this knowledge to LLM. These three operations improve the accuracy of LLM for complex knowledge-intensive tasks in terms of reasoning ability and knowledge. Finally, SearChain generates the reasoning process and marks references to supporting documents for each reasoning step, which improves traceability. SearChain transforms the topology of reasoning from chain to tree, which can modify the reasoning direction. Experiment shows that SearChain outperforms baselines on complex knowledge-intensive tasks including multi-hop question-answering, slot filling, fact checking, and long-form question-answering.

翻译：使ChatGPT等大型语言模型生成的内容准确、可信且可追溯至关重要，尤其在需要多步推理且每步都需要知识支撑的复杂知识密集型任务中。引入信息检索为语言模型提供外部知识是解决该问题的可行方案，但如何在语言模型中引入信息检索以及引入位置仍是巨大挑战。现有工作存在检索到的错误知识误导语言模型或打断其推理链的问题。本文提出名为Search-in-the-Chain（SearChain）的新框架，通过语言模型与检索系统的协同交互解决上述挑战。首先，语言模型生成全局推理链——查询链（Chain-of-Query, CoQ），其中每个节点包含面向检索的查询及其对应答案。其次，检索系统验证CoQ各节点答案，当检索结果置信度较高时修正与检索信息不一致的答案，从而提升可信度。第三，语言模型可在CoQ中标记缺失知识，检索系统可为其补充相应知识。这三项操作通过增强推理能力与知识完备性，显著提升了语言模型在复杂知识密集型任务中的准确性。最终，SearChain生成推理过程并为每个推理步骤标注支持文档引用，实现可追溯性。该框架将推理拓扑结构从链式扩展为树状，可灵活修正推理方向。实验表明，在包含多跳问答、槽填充、事实核查和长文本问答的复杂知识密集型任务中，SearChain性能全面超越基线模型。