With the increasing capabilities of Large Language Models (LLMs), parallel reasoning has emerged as a new inference paradigm that enhances reasoning robustness by concurrently exploring multiple lines of thought before converging on a final answer. It has become a significant trend to explore parallel reasoning to overcome the fragility of standard sequential methods and improve practical performance. In this paper, we aim to survey and summarize the progress and challenges of parallel reasoning. We first present a formal definition of parallel reasoning and clarify its distinction from related concepts like Chain-of-Thought. Then, we organize and discuss advanced techniques based on a novel taxonomy, including non-interactive reasoning, interactive reasoning, and efficiency-focused decoding strategies. Additionally, we explore various application scenarios, such as solving complex problems and enhancing the reliability of LLM outputs.Finally, we highlight the core challenges of parallel reasoning and suggest potential directions for future research. We hope that our work can provide a useful roadmap for beginners and encourage more research on improving parallel reasoning methods. Related source can be avaliable in https://github.com/PPPP-kaqiu/Awesome-Parallel-Reasoning.

翻译：随着大语言模型（LLM）能力的不断提升，并行推理作为一种新兴的推理范式应运而生。它通过同时探索多条思路并在最终收敛前进行整合，从而增强了推理的鲁棒性。探索并行推理以克服标准顺序方法的脆弱性并提升实际性能，已成为一个重要趋势。本文旨在系统梳理和总结并行推理的研究进展与挑战。我们首先给出并行推理的形式化定义，并厘清其与思维链等相关概念的区别。随后，我们基于一种新颖的分类法（包括非交互式推理、交互式推理以及注重效率的解码策略）对前沿技术进行了梳理与讨论。此外，我们还探讨了并行推理在解决复杂问题和提升LLM输出可靠性等多种应用场景中的作用。最后，我们重点指出了并行推理面临的核心挑战，并展望了未来潜在的研究方向。我们希望本工作能为初学者提供一份有用的路线图，并激励更多研究致力于改进并行推理方法。相关资源可在 https://github.com/PPPP-kaqiu/Awesome-Parallel-Reasoning 获取。