Making the contents generated by Large Language Model (LLM) such as ChatGPT, accurate, credible and traceable is crucial, especially in complex knowledge-intensive tasks that require multi-step reasoning and each of which needs knowledge to solve. Introducing Information Retrieval (IR) to provide LLM with external knowledge is good potential to solve this problem. However, where and how to introduce IR into LLM is a big challenge. Previous work has the disadvantage that the wrong knowledge retrieved by IR misleads the LLM or breaks the reasoning chain of LLM. In this paper, we propose a novel framework called Search-in-the-Chain (SearChain) for the interaction between LLM and IR to solve the challenges. First, LLM generates the global reasoning chain called Chain-of-Query (CoQ) where each node consists of an IR-oriented query and the answer to the query. Second, IR verifies the answer of each node of CoQ, it corrects the answer that is not consistent with the retrieved information when IR gives high confidence, which improves the credibility. Third, LLM can mark its missing knowledge in CoQ and IR can provide this knowledge to LLM. These three operations improve the accuracy of LLM for complex knowledge-intensive tasks in terms of reasoning ability and knowledge. Finally, SearChain generates the reasoning process and marks references to supporting documents for each reasoning step, which improves traceability. SearChain transforms the topology of reasoning from chain to tree, which can modify the reasoning direction. Experiment shows that SearChain outperforms baselines on complex knowledge-intensive tasks including multi-hop question-answering, slot filling, fact checking, and long-form question-answering.

翻译：使大语言模型（如ChatGPT）生成的内容准确、可信且可溯源至关重要，尤其在需要多步推理且每一步需知识支持的知识密集型复杂任务中。引入信息检索为LLM提供外部知识是解决该问题的可行方案，但如何及在何处将IR融入LLM仍是重大挑战。现有工作存在检索错误知识误导LLM或破坏其推理链的缺陷。本文提出名为Search-in-the-Chain（SearChain）的新型框架，通过LLM与IR的交互解决上述挑战。首先，LLM生成全局推理链（Chain-of-Query, CoQ），其每个节点包含面向IR的查询及对应答案；其次，IR验证CoQ各节点答案，当IR置信度较高时修正与检索信息不符的答案，增强可信性；第三，LLM可标记CoQ中的知识缺失，由IR提供所需知识。这三项操作从推理能力与知识层面提升了LLM处理复杂知识密集型任务的准确性。最终，SearChain生成推理过程并为每个推理步骤标注支持文档的引用，增强可溯源性。SearChain将推理拓扑从链式结构转化为树状结构，可修正推理方向。实验表明，SearChain在包括多跳问答、槽填充、事实核查及长文本问答在内的复杂知识密集型任务中均优于基线方法。