Robotic vision applications often necessitate a wide range of visual perception tasks, such as object detection, segmentation, and identification. While there have been substantial advances in these individual tasks, integrating specialized models into a unified vision pipeline presents significant engineering challenges and costs. Recently, Multimodal Large Language Models (MLLMs) have emerged as novel backbones for various downstream tasks. We argue that leveraging the pre-training capabilities of MLLMs enables the creation of a simplified framework, thus mitigating the need for task-specific encoders. Specifically, the large-scale pretrained knowledge in MLLMs allows for easier fine-tuning to downstream robotic vision tasks and yields superior performance. We introduce the RoboLLM framework, equipped with a BEiT-3 backbone, to address all visual perception tasks in the ARMBench challenge-a large-scale robotic manipulation dataset about real-world warehouse scenarios. RoboLLM not only outperforms existing baselines but also substantially reduces the engineering burden associated with model selection and tuning. The source code is publicly available at https://github.com/longkukuhi/armbench.

翻译：[translated abstract in Chinese] 机器人视觉应用通常需要一系列广泛的视觉感知任务，例如目标检测、分割与识别。尽管这些独立任务已取得显著进展，将专用模型整合到统一视觉流水线中仍面临巨大的工程挑战与成本。近期，多模态大语言模型（MLLMs）作为新型骨干网络涌现，可服务于各类下游任务。我们提出，利用MLLMs的预训练能力能够构建简化框架，从而避免对任务特定编码器的需求。具体而言，MLLMs中大规模预训练知识可更轻松地微调至下游机器人视觉任务，并产生更优性能。我们介绍了配备BEiT-3骨干网络的RoboLLM框架，用于解决ARMBench挑战中的所有视觉感知任务——该挑战是一个关于真实仓储场景的大规模机器人操作数据集。RoboLLM不仅超越了现有基线模型，还大幅降低了模型选择与调优带来的工程负担。源代码已公开发布于 https://github.com/longkukuhi/armbench。