Chain-of-thought (CoT) reasoning sometimes fails to faithfully reflect the true computation of a large language model (LLM), hampering its utility in explaining how LLMs arrive at their answers. Moreover, optimizing for faithfulness and interpretability in reasoning often degrades task performance. To address this tradeoff and improve CoT faithfulness, we propose Reasoning Execution by Multiple Listeners (REMUL), a multi-party reinforcement learning approach. REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful. A speaker model generates a reasoning trace, which is truncated and passed to a pool of listener models who "execute" the trace, continuing the trace to an answer. Speakers are rewarded for producing reasoning that is clear to listeners, with additional correctness regularization via masked supervised finetuning to counter the tradeoff between faithfulness and performance. On multiple reasoning benchmarks (BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO), REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy. Our analysis finds that these gains are robust across training domains, translate to legibility gains, and are associated with shorter and more direct CoTs.

翻译：思维链推理有时无法忠实反映大型语言模型的实际计算过程，这削弱了其在解释模型如何得出答案方面的效用。此外，优化推理过程的忠实性与可解释性往往会降低任务性能。为解决这一权衡问题并提升思维链的忠实度，我们提出多听众推理执行方法——一种基于多方强化学习的技术。该方法建立在以下假设之上：能够被其他方理解的推理轨迹将具有更高的忠实性。发言者模型生成推理轨迹，该轨迹经过截断后传递给听众模型池，由听众"执行"该轨迹以延续推理直至得出答案。发言者因生成清晰易懂的推理而获得奖励，同时通过掩码监督微调进行正确性正则化，以平衡忠实性与性能之间的权衡。在多个推理基准测试中，该方法持续显著提升了三项忠实性指标——提示归因、早期回答曲线下面积和错误注入曲线下面积——同时提高了准确率。我们的分析表明，这些改进在不同训练领域具有鲁棒性，能够转化为可读性提升，且与更简短直接的思维链相关。