Enhancing the reasoning capabilities of large language models (LLMs) remains a key challenge, especially for tasks that require complex, multi-step decision-making. Humans excel at these tasks by leveraging deliberate planning with an internal world model to simulate the potential outcomes of various actions. Inspired by this, we propose a novel multi-step reasoning framework for LLMs, referred to as Structure-aware Planning with Accurate World Model (SWAP). Unlike previous approaches that rely solely on Chain-of-Thought (CoT) reasoning in natural language, SWAP incorporates structural information to guide the reasoning process via a world model and provides a soft verification mechanism over the steps. Moreover, SWAP overcomes the challenge of accurate world state predictions in complex reasoning tasks by introducing a Generator-Discriminator architecture, which enables more reliable world modeling. Specifically, the generator predicts the next state, and the discriminator ensures alignment with the logical consistency required by the problem context. SWAP also encourages the policy model to explore a broad range of potential actions to prevent premature convergence. By resolving the bottlenecks of generation diversity for both actions and states using diversity-based modeling (DBM) and improving discrimination accuracy through contrastive ranking (CR), SWAP significantly enhances the reasoning performance of LLMs. We evaluate SWAP across diverse reasoning-intensive benchmarks including math reasoning, logical reasoning, and coding tasks. Extensive experiments demonstrate that SWAP achieves substantial improvements over the baselines and consistently outperforms existing methods.

翻译：提升大语言模型（LLMs）的推理能力仍是一个关键挑战，尤其对于需要复杂多步决策的任务。人类在这些任务中表现出色，其通过利用内部世界模型进行审慎规划来模拟各种行动的潜在结果。受此启发，我们提出了一种新颖的用于LLMs的多步推理框架，称为基于精确世界模型的结构感知规划（SWAP）。与以往仅依赖自然语言思维链（CoT）推理的方法不同，SWAP通过世界模型融入结构信息以指导推理过程，并提供对推理步骤的软验证机制。此外，SWAP通过引入生成器-判别器架构克服了复杂推理任务中世界状态预测不精确的挑战，从而实现了更可靠的世界建模。具体而言，生成器预测下一状态，判别器则确保其与问题上下文所要求的逻辑一致性保持一致。SWAP还鼓励策略模型探索广泛的潜在行动以防止过早收敛。通过使用基于多样性的建模（DBM）解决行动和状态生成多样性的瓶颈，并通过对比排序（CR）提高判别准确性，SWAP显著提升了LLMs的推理性能。我们在包括数学推理、逻辑推理和编码任务在内的多种推理密集型基准上评估SWAP。大量实验表明，SWAP相较于基线方法取得了显著提升，并持续优于现有方法。