Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robotic hardware for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.

翻译：三个挑战限制了机器人学习研究的进展：机器人成本高昂（只有少数实验室能够参与）、不同实验室使用不同的机器人（研究结果无法跨实验室泛化），以及我们缺乏互联网规模的机器人数据。我们通过一个新的基准来应对这些挑战：离线训练，在线测试（TOTO）。TOTO为远程用户提供对共享机器人硬件的访问权限，以便在共同任务上评估方法，并提供这些任务的开源数据集用于离线训练。其操作任务套件要求对未见过的物体、位置和光照条件具有挑战性的泛化能力。我们展示了TOTO的初步结果，比较了五种预训练视觉表示和四种离线策略学习基线，这些成果由五个机构远程贡献。然而，TOTO的真正前景在于未来：我们发布此基准以供任何用户提交额外结果，从而无需获取硬件或收集数据，即可轻松、直接地与多种方法进行比较。