Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning for tasks with inputs of any form. Our framework incorporates visual planning to capture intricate environmental details, while language planning enhances the logical coherence of the overall system. We evaluate the effectiveness of our framework across vision-language tasks, vision-only tasks, and language-only tasks. The results demonstrate the superior performance of our approach, indicating that the integration of visual and language planning yields better contextually aware task execution.

翻译：大型语言模型（LLMs）和大型多模态模型（LMMs）在各种任务中展现出卓越的决策掩码能力。然而，它们本质上是在语言空间内进行规划，缺乏视觉与空间想象能力。相比之下，人类在思考过程中会同时使用左右脑半球进行语言与视觉规划。因此，本文提出了一种新颖的视觉-语言规划框架，对任意形式输入的任务同时进行视觉与语言规划。该框架通过视觉规划捕捉复杂的环境细节，而语言规划则增强整体系统的逻辑连贯性。我们在视觉-语言任务、纯视觉任务和纯语言任务上评估了框架的有效性。结果表明，我们的方法具有优越性能，验证了视觉与语言规划的融合能实现更具上下文感知的任务执行。