Recent works such as VisProg and ViperGPT have smartly composed foundation models for visual reasoning-using large language models (LLMs) to produce programs that can be executed by pre-trained vision-language models. However, they operate in limited domains, such as 2D images, not fully exploiting the generalization of language: abstract concepts like "left" can also be grounded in 3D, temporal, and action data, as in moving to your left. This limited generalization stems from these inference-only methods' inability to learn or adapt pre-trained models to a new domain. We propose the Logic-Enhanced Foundation Model (LEFT), a unified framework that learns to ground and reason with concepts across domains with a differentiable, domain-independent, first-order logic-based program executor. LEFT has an LLM interpreter that outputs a program represented in a general, logic-based reasoning language, which is shared across all domains and tasks. LEFT's executor then executes the program with trainable domain-specific grounding modules. We show that LEFT flexibly learns concepts in four domains: 2D images, 3D scenes, human motions, and robotic manipulation. It exhibits strong reasoning ability in a wide variety of tasks, including those that are complex and not seen during training, and can be easily applied to new domains.

翻译：近期的工作如VisProg和ViperGPT，通过使用大型语言模型生成程序，并由预训练的视觉-语言模型执行，巧妙地组合了基础模型以进行视觉推理。然而，它们仅在有限领域（如二维图像）中运行，未能充分利用语言的泛化能力：像“左”这样的抽象概念也可以在三围、时间性和动作数据中接地，例如“向左移动”。这种有限的泛化源于这些仅推断的方法无法学习或调整预训练模型以适应新领域。我们提出逻辑增强基础模型（LEFT），这是一个统一框架，通过可微分、领域无关、基于一阶逻辑的程序执行器，学习跨领域的概念接地和推理。LEFT有一个LLM解释器，输出以通用的、基于逻辑的推理语言表示的程序，该语言在所有领域和任务间共享。LEFT的执行器随后使用可训练的领域特定接地模块执行程序。我们展示了LEFT在四个领域中灵活学习概念：二维图像、三维场景、人体运动和机器人操作。它在各种任务中表现出强大的推理能力，包括复杂且训练中未见过的任务，并能轻松应用于新领域。