When prompted with a few examples and intermediate steps, large language models (LLMs) have demonstrated impressive performance in various reasoning tasks. However, prompting methods that rely on implicit knowledge in an LLM often generate incorrect answers when the implicit knowledge is wrong or inconsistent with the task. To tackle this problem, we present Hypotheses-to-Theories (HtT), a framework that learns a rule library for reasoning with LLMs. HtT contains two stages, an induction stage and a deduction stage. In the induction stage, an LLM is first asked to generate and verify rules over a set of training examples. Rules that appear and lead to correct answers sufficiently often are collected to form a rule library. In the deduction stage, the LLM is then prompted to employ the learned rule library to perform reasoning to answer test questions. Experiments on relational reasoning, numerical reasoning and concept learning problems show that HtT improves existing prompting methods, with an absolute gain of 10-30% in accuracy. The learned rules are also transferable to different models and to different forms of the same problem.

翻译：当通过少量示例和中间步骤进行提示时，大型语言模型（LLM）在各种推理任务中展现出了令人瞩目的性能。然而，依赖LLM内隐知识的提示方法，当内隐知识存在错误或与任务不一致时，往往会产生不正确答案。为解决此问题，我们提出“假设到理论”（HtT）框架，该框架通过学习规则库来辅助LLM进行推理。HtT包含两个阶段：归纳阶段和演绎阶段。在归纳阶段，首先让LLM在一组训练示例上生成并验证规则；那些频繁出现且能导向正确答案的规则被收集起来，构成规则库。在演绎阶段，再利用该规则库提示LLM进行推理，以回答测试问题。在关系推理、数值推理和概念学习问题上的实验表明，HtT改进了现有提示方法，准确率绝对提升10%至30%。所学习的规则还可迁移至不同模型以及同一问题的不同形式。