Retrieval-Augmented Generation (RAG), by incorporating external knowledge with parametric memory of language models, has become the state-of-the-art architecture for open-domain QA tasks. However, common knowledge bases are inherently constrained by limited coverage and noisy information, making retrieval-based approaches inadequate to answer implicit reasoning questions. In this paper, we propose an Induction-Augmented Generation (IAG) framework that utilizes inductive knowledge along with the retrieved documents for implicit reasoning. We leverage large language models (LLMs) for deriving such knowledge via a novel prompting method based on inductive reasoning patterns. On top of this, we implement two versions of IAG named IAG-GPT and IAG-Student, respectively. IAG-GPT directly utilizes the knowledge generated by GPT-3 for answer prediction, while IAG-Student gets rid of dependencies on GPT service at inference time by incorporating a student inductor model. The inductor is firstly trained via knowledge distillation and further optimized by back-propagating the generator feedback via differentiable beam scores. Experimental results show that IAG outperforms RAG baselines as well as ChatGPT on two Open-Domain QA tasks. Notably, our best models have won the first place in the official leaderboards of CSQA2.0 (since Nov 1, 2022) and StrategyQA (since Jan 8, 2023).

翻译：检索增强生成（RAG）通过将语言模型的参数化记忆与外部知识相结合，已成为开放域问答任务的最先进架构。然而，常见知识库固有的覆盖范围有限和噪声信息问题，使得基于检索的方法难以回答隐含推理问题。本文提出归纳增强生成（IAG）框架，利用归纳知识与检索文档进行隐含推理。我们借助大型语言模型（LLM），通过基于归纳推理模式的新型提示方法来推导此类知识。在此基础上，我们实现了IAG的两个版本，分别称为IAG-GPT和IAG-Student。IAG-GPT直接利用GPT-3生成的知识进行答案预测，而IAG-Student通过引入学生归纳模型，在推理阶段摆脱了对GPT服务的依赖。该归纳模型首先通过知识蒸馏进行训练，然后通过可微分的波束分数反向传播生成器反馈进行进一步优化。实验结果表明，IAG在两个开放域问答任务上优于RAG基线以及ChatGPT。值得注意的是，我们的最佳模型在CSQA2.0（自2022年11月1日起）和StrategyQA（自2023年1月8日起）的官方排行榜上均获得第一名。