Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train generalist X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. More details can be found on the project website $\href{https://robotics-transformer-x.github.io}{\text{robotics-transformer-x.github.io}}$.

翻译：大规模、高容量模型在多样化数据集上训练后，在高效处理下游应用方面展现出显著成功。从自然语言处理到计算机视觉等领域，这导致了预训练模型的整合，通用预训练主干成为许多应用的起点。机器人领域能否实现这种整合？传统上，机器人学习方法为每个应用、每个机器人，甚至每个环境分别训练单独模型。我们能否训练通用的X-机器人策略，使其能高效适应新机器人、新任务和新环境？本文提供了标准化数据格式的数据集和模型，以在机器人操作背景下探索这一可能性，同时提供实验结果为有效的X-机器人策略提供示例。我们通过21家机构合作，从22种不同机器人收集了数据集，展示了527项技能（160266个任务）。我们证明，在此数据上训练的高容量模型——我们称之为RT-X——表现出正向迁移，并通过利用其他平台的经验提升了多种机器人的能力。更多详情可访问项目网站$\href{https://robotics-transformer-x.github.io}{\text{robotics-transformer-x.github.io}}$。