LLMs have shown impressive progress in robotics (e.g., manipulation and navigation) with natural language task descriptions. The success of LLMs in these tasks leads us to wonder: What is the ability of LLMs to solve vehicle routing problems (VRPs) with natural language task descriptions? In this work, we study this question in three steps. First, we construct a dataset with 21 types of single- or multi-vehicle routing problems. Second, we evaluate the performance of LLMs across four basic prompt paradigms of text-to-code generation, each involving different types of text input. We find that the basic prompt paradigm, which generates code directly from natural language task descriptions, performs the best for GPT-4, achieving 56% feasibility, 40% optimality, and 53% efficiency. Third, based on the observation that LLMs may not be able to provide correct solutions at the initial attempt, we propose a framework that enables LLMs to refine solutions through self-reflection, including self-debugging and self-verification. With GPT-4, our proposed framework achieves a 16% increase in feasibility, a 7% increase in optimality, and a 15% increase in efficiency. Moreover, we examine the sensitivity of GPT-4 to task descriptions, specifically focusing on how its performance changes when certain details are omitted from the task descriptions, yet the core meaning is preserved. Our findings reveal that such omissions lead to a notable decrease in performance: 4% in feasibility, 4% in optimality, and 5% in efficiency. Website: https://sites.google.com/view/words-to-routes/

翻译：大语言模型在机器人领域（如操控与导航）的基于自然语言任务描述的任务上取得了显著进展。这不禁让我们思考：大语言模型在解决带有自然语言任务描述的车辆路径规划问题时能力如何？本研究分三步探讨该问题。首先，我们构建了一个包含21类单车或多车路径规划问题的数据集。其次，我们评估了四种基础文本到代码生成提示范式下大语言模型的性能，这些范式涉及不同类型的文本输入。结果表明，直接根据自然语言任务描述生成代码的基础提示范式对GPT-4表现最佳，实现了56%的可行性、40%的最优性和53%的效率。第三，基于大语言模型可能无法在初次尝试中提供正确解决方案的观察，我们提出了一种框架，使大语言模型能够通过自我反思（包括自我调试与自我验证）优化解决方案。在GPT-4上，该框架使可行性提升16%，最优性提升7%，效率提升15%。此外，我们检验了GPT-4对任务描述的敏感性，重点关注在保留核心含义的前提下，当任务描述省略某些细节时其性能的变化。研究发现此类省略导致性能显著下降：可行性降低4%，最优性降低4%，效率降低5%。网址：https://sites.google.com/view/words-to-routes/