Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robotic hardware for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.

翻译：三个挑战制约了机器人学习研究的进展：机器人成本高昂（少数实验室能够参与）、各实验室使用不同机器人（研究结果无法跨实验室推广），以及缺乏互联网规模的机器人数据。我们通过一个新的基准——离线训练，在线测试（TOTO）——来应对这些挑战。TOTO为远程用户提供共享机器人硬件访问权限，用于在共同任务上评估方法，同时提供这些任务的开源数据集供离线训练。其操作任务套件要求对未见过的物体、位置和光照条件进行具有挑战性的泛化。我们展示了TOTO的初步结果，比较了五种预训练视觉表示和四种离线策略学习基线，这些成果由五个机构远程贡献。然而，TOTO的真正潜力在于未来：我们开放该基准供任何用户提交额外评测结果，无需获取硬件或收集数据即可轻松实现与多种方法的直接比较。