Despite recent success in large language model (LLM) reasoning, LLMs struggle with hierarchical multi-step reasoning tasks like generating complex programs. For these tasks, humans often start with a high-level algorithmic design and implement each part gradually. We introduce Parsel, a framework enabling automatic implementation and validation of complex algorithms with code LLMs, taking hierarchical function descriptions in natural language as input. We show that Parsel can be used across domains requiring hierarchical reasoning, including program synthesis, robotic planning, and theorem proving. We show that LLMs generating Parsel solve more competition-level problems in the APPS dataset, resulting in pass rates that are over 75% higher than prior results from directly sampling AlphaCode and Codex, while often using a smaller sample budget. We also find that LLM-generated robotic plans using Parsel as an intermediate language are more than twice as likely to be considered accurate than directly generated plans. Lastly, we explore how Parsel addresses LLM limitations and discuss how Parsel may be useful for human programmers.

翻译：尽管大语言模型（LLM）在推理方面近期取得了成功，但LLM在应对层级式多步推理任务（如生成复杂程序）时仍存在困难。针对此类任务，人类通常从高层算法设计入手，再逐步实现各个部分。我们提出Parsel框架，该框架使代码LLM能够自动实现并验证复杂算法，以自然语言描述的层级函数说明作为输入。研究表明，Parsel可应用于需要层级推理的多个领域，包括程序合成、机器人规划及定理证明。我们发现，使用Parsel生成代码的LLM在APPS数据集中解决了更多竞赛级问题，其通过率相比直接采样AlphaCode和Codex的先前结果提升超过75%，且通常使用更少的采样预算。同时，采用Parsel作为中间语言生成的机器人规划方案，其准确率是直接生成规划的两倍以上。最后，我们探讨了Parsel如何弥补LLM的局限性，并讨论该框架对人类程序员的潜在价值。