Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.

翻译：近期视觉语言基础模型的进展表明，它们能够理解多模态数据并解决复杂的视觉语言任务，包括机器人操作。我们寻求一种直接利用现有视觉语言模型（VLM）的方法，通过简单的微调应用于机器人数据。为此，我们推导出一个简单新颖的视觉语言操作框架，名为RoboFlamingo，它基于开源VLM——OpenFlamingo构建。与以往工作不同，RoboFlamingo利用预训练VLM进行单步视觉语言理解，通过显式策略头建模序列历史信息，并仅通过语言条件操作数据集的模仿学习进行轻微微调。这种分解为RoboFlamingo提供了开环控制的灵活性，并可在低性能平台上部署。通过在测试基准上大幅超越现有最优性能，我们证明RoboFlamingo可以成为将VLM适配到机器人控制的有效且具有竞争力的替代方案。我们广泛的实验结果还揭示了关于不同预训练VLM在操作任务中行为的一些有趣结论。我们相信RoboFlamingo有潜力成为机器人操作中低成本且易用的解决方案，使每个人都能微调自己的机器人策略。