Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website https://robotics-transformer-x.github.io.

翻译：在多样化数据集上训练的大型高容量模型已展现出高效处理下游应用的显著成功。从自然语言处理到计算机视觉等领域，这导致了预训练模型的整合——通用预训练主干网络成为众多应用的起点。此类整合能否在机器人学领域实现？传统上，机器人学习方法需为每个应用、每台机器人甚至每种环境单独训练模型。我们能否转而训练通用的X-机器人策略，使其能高效适配新的机器人、任务和环境？本文通过提供标准化数据格式的数据集与模型，使在机器人操控领域探索这种可能性成为可能，同时提供展示有效X-机器人策略的实证结果。我们通过21个机构合作汇集了来自22种机器人的数据集，涵盖527项技能（160266个任务）。研究表明，基于此数据训练的高容量模型（我们称之为RT-X）展现出正向迁移特性，能通过融合其他平台的实践经验提升多种机器人的能力。更多细节详见项目网站 https://robotics-transformer-x.github.io。