Large Language Models (LLMs) equipped with extensive world knowledge and strong reasoning skills can tackle diverse tasks across domains, often by posing them as conversation-style instruction-response pairs. In this paper, we propose LLaRA: Large Language and Robotics Assistant, a framework which formulates robot action policy as conversations, and provides improved responses when trained with auxiliary data that complements policy learning. LLMs with visual inputs, i.e., Vision Language Models (VLMs), have the capacity to process state information as visual-textual prompts and generate optimal policy decisions in text. To train such action policy VLMs, we first introduce an automated pipeline to generate diverse high-quality robotics instruction data from existing behavior cloning data. A VLM finetuned with the resulting collection of datasets based on a conversation-style formulation tailored for robotics tasks, can generate meaningful robot action policy decisions. Our experiments across multiple simulated and real-world environments demonstrate the state-of-the-art performance of the proposed LLaRA framework. The code, datasets, and pretrained models are available at https://github.com/LostXine/LLaRA.

翻译：大型语言模型（LLMs）具备广泛的世界知识和强大的推理能力，通常通过构建对话式指令-响应对来处理跨领域的多样化任务。本文提出LLaRA：大型语言与机器人助手，该框架将机器人动作策略构建为对话形式，并通过利用辅助数据增强策略学习来提供更优的响应。具备视觉输入能力的LLMs，即视觉语言模型（VLMs），能够将状态信息处理为视觉-文本提示并生成最优的文本策略决策。为训练此类动作策略VLMs，我们首先提出一种自动化流程，可从现有行为克隆数据中生成多样化、高质量的机器人指令数据。基于专为机器人任务设计的对话式框架，使用由此生成的数据集集合进行微调的VLM能够生成有意义的机器人动作策略决策。我们在多个仿真与真实环境中的实验证明了所提出的LLaRA框架具备最先进的性能。代码、数据集及预训练模型公开于https://github.com/LostXine/LLaRA。