As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics. By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior.

翻译：随着大语言模型执行越来越复杂的任务，验证其行为的正确性和安全性变得愈加困难。一种应对方法是提示模型外化其推理过程，例如在回答问题时生成逐步推理（思维链，CoT）。这种推理过程使我们能够检查模型执行任务所采用的流程。然而，这种方法依赖于表述推理能忠实反映模型的实际推理，而实际情况并非总是如此。为改进CoT推理的忠实性，我们让模型通过将问题分解为子问题来生成推理。基于分解的方法在问答任务上取得了强劲表现，有时接近CoT的效果，同时在多个最近提出的评估指标上提升了模型表述推理的忠实性。通过强制模型在独立上下文中回答更简单的子问题，我们显著提高了模型生成推理相比CoT的忠实性，同时仍保留部分CoT的性能增益。我们的结果表明，提升模型生成推理的忠实性是可行的；持续改进可能使推理过程最终能用于验证LLM行为的正确性与安全性。