Making the contents generated by Large Language Model (LLM) such as ChatGPT, accurate, credible and traceable is crucial, especially in complex knowledge-intensive tasks that require multi-step reasoning and each of which needs knowledge to solve. Introducing Information Retrieval (IR) to provide LLM with external knowledge is good potential to solve this problem. However, where and how to introduce IR into LLM is a big challenge. Previous work has the disadvantage that the wrong knowledge retrieved by IR misleads the LLM or breaks the reasoning chain of LLM. In this paper, we propose a novel framework called Search-in-the-Chain (SearChain) for the interaction between LLM and IR to solve the challenges. First, LLM generates the global reasoning chain called Chain-of-Query (CoQ) where each node consists of an IR-oriented query and the answer to the query. Second, IR verifies the answer of each node of CoQ, it corrects the answer that is not consistent with the retrieved information when IR gives high confidence, which improves the credibility. Third, LLM can mark its missing knowledge in CoQ and IR can provide this knowledge to LLM. These three operations improve the accuracy of LLM for complex knowledge-intensive tasks in terms of reasoning ability and knowledge. Finally, SearChain generates the reasoning process and marks references to supporting documents for each reasoning step, which improves traceability. SearChain transforms the topology of reasoning from chain to tree, which can modify the reasoning direction. Experiment shows that SearChain outperforms baselines on complex knowledge-intensive tasks including multi-hop question-answering, slot filling, fact checking, and long-form question-answering.

翻译：摘要：使ChatGPT等大语言模型（LLM）生成的内容准确、可信且可追溯至关重要，尤其是在需要多步推理且每一步均需领域知识的复杂知识密集型任务中。引入信息检索（IR）为LLM提供外部知识是解决该问题的有效途径，但如何以及何时将IR融入LLM仍面临重大挑战。现有工作存在检索错误知识误导LLM或破坏其推理链的缺陷。本文提出名为Search-in-the-Chain（SearChain）的新框架，通过LLM与IR的协同交互应对上述挑战。首先，LLM生成全局推理链——查询链（CoQ），其中每个节点包含面向IR的查询及其对应答案。其次，IR验证CoQ各节点的答案，当检索信息高度置信时修正与检索结果不一致的答案，从而提升可信度。再次，LLM可在CoQ中标记缺失知识并由IR补充，这三项操作从推理能力与知识维度提升LLM在复杂知识密集型任务中的准确性。最后，SearChain生成推理过程并为每个推理步骤标注支持文档的引用，增强可追溯性。该框架将推理拓扑结构从链式转变为树状，从而修正推理方向。实验表明，SearChain在多跳问答、槽填充、事实核查和长篇问答等复杂知识密集型任务中均优于基线方法。