Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.

翻译：近期视觉语言基础模型的进展展示了它们理解多模态数据并解决复杂视觉语言任务（包括机器人操作）的能力。我们寻求一种直接利用现有视觉语言模型（VLM）的简单方法，仅需对机器人数据进行微调。为此，我们基于开源VLM——OpenFlamingo，推导出一个简洁新颖的视觉语言操作框架，命名为RoboFlamingo。与先前工作不同，RoboFlamingo利用预训练VLM进行单步视觉语言理解，通过显式策略头对序列历史信息进行建模，并仅通过模仿学习在语言条件操作数据集上进行轻量微调。这种分解使得RoboFlamingo在开环控制及低性能平台部署方面具备灵活性。通过在测试基准上以显著优势超越当前最优性能，我们证明RoboFlamingo是将VLM适配至机器人控制的有效且具竞争力的替代方案。广泛实验还揭示了不同预训练VLM在操作任务中行为的若干有趣结论。我们相信RoboFlamingo有潜力成为机器人操作中高性价比且易于使用的解决方案，使每个人都能微调自己的机器人策略。