When prompted with a few examples and intermediate steps, large language models (LLMs) have demonstrated impressive performance in various reasoning tasks. However, prompting methods that rely on implicit knowledge in an LLM often hallucinate incorrect answers when the implicit knowledge is wrong or inconsistent with the task. To tackle this problem, we present Hypotheses-to-Theories (HtT), a framework that learns a rule library for reasoning with LLMs. HtT contains two stages, an induction stage and a deduction stage. In the induction stage, an LLM is first asked to generate and verify rules over a set of training examples. Rules that appear and lead to correct answers sufficiently often are collected to form a rule library. In the deduction stage, the LLM is then prompted to employ the learned rule library to perform reasoning to answer test questions. Experiments on both numerical reasoning and relational reasoning problems show that HtT improves existing prompting methods, with an absolute gain of 11-27% in accuracy. The learned rules are also transferable to different models and to different forms of the same problem.

翻译：当通过少量示例和中间步骤进行提示时，大语言模型（LLM）已在各类推理任务中展现出令人瞩目的性能。然而，依赖LLM内隐知识的提示方法在隐含知识错误或与任务不一致时，往往会产生幻觉性错误答案。为解决这一问题，我们提出假设到理论（HtT）框架，该框架通过学习规则库实现LLM推理。HtT包含归纳与演绎两个阶段：归纳阶段首先让LLM对训练示例生成并验证规则，将频繁出现且导向正确结果的规则收录至规则库；演绎阶段则引导LLM运用所学规则库进行推理以回答测试问题。在数值推理与关系推理问题上的实验表明，HtT显著改进了现有提示方法，准确率绝对提升达11%-27%。此外，所学规则可迁移至不同模型及同一问题的不同表现形式。