Current vision-based robotics simulation benchmarks have significantly advanced robotic manipulation research. However, robotics is fundamentally a real-world problem, and evaluation for real-world applications has lagged behind in evaluating generalist policies. In this paper, we discuss challenges and desiderata in designing benchmarks for generalist robotic manipulation policies for the goal of sim-to-real policy transfer. We propose 1) utilizing high visual-fidelity simulation for improved sim-to-real transfer, 2) evaluating policies by systematically increasing task complexity and scenario perturbation to assess robustness, and 3) quantifying performance alignment between real-world performance and its simulation counterparts.

翻译：当前基于视觉的机器人仿真基准显著推动了机器人操作研究的发展。然而，机器人学本质上是一个现实世界问题，面向实际应用的评估在通用策略评估方面仍显滞后。本文围绕仿真到现实策略迁移的目标，探讨了设计通用机器人操作策略基准所面临的挑战与需求。我们提出：1）利用高视觉保真度仿真以提升仿真到现实迁移效果；2）通过系统性增加任务复杂性和场景扰动来评估策略的鲁棒性；3）量化现实世界性能与其仿真对应表现之间的对齐程度。