As a representative cyber-physical system (CPS), robotic manipulator has been widely adopted in various academic research and industrial processes, indicating its potential to act as a universal interface between the cyber and the physical worlds. Recent studies in robotics manipulation have started employing artificial intelligence (AI) approaches as controllers to achieve better adaptability and performance. However, the inherent challenge of explaining AI components introduces uncertainty and unreliability to these AI-enabled robotics systems, necessitating a reliable development platform for system design and performance assessment. As a foundational step towards building reliable AI-enabled robotics systems, we propose a public industrial benchmark for robotics manipulation in this paper. It leverages NVIDIA Omniverse Isaac Sim as the simulation platform, encompassing eight representative manipulation tasks and multiple AI software controllers. An extensive evaluation is conducted to analyze the performance of AI controllers in solving robotics manipulation tasks, enabling a thorough understanding of their effectiveness. To further demonstrate the applicability of our benchmark, we develop a falsification framework that is compatible with physical simulators and OpenAI Gym environments. This framework bridges the gap between traditional testing methods and modern physics engine-based simulations. The effectiveness of different optimization methods in falsifying AI-enabled robotics manipulation with physical simulators is examined via a falsification test. Our work not only establishes a foundation for the design and development of AI-enabled robotics systems but also provides practical experience and guidance to practitioners in this field, promoting further research in this critical academic and industrial domain.

翻译：作为网络-物理系统（CPS）的代表，机器人操作臂已广泛应用于各类学术研究与工业流程中，展现出作为网络空间与物理世界通用接口的潜力。近期机器人操作领域的研究开始采用人工智能（AI）方法作为控制器，以提升系统的适应性与性能。然而，AI组件固有的可解释性难题为这些AI赋能的机器人系统带来了不确定性与不可靠性，亟需一个可靠的开发生态系统用于系统设计与性能评估。作为构建可靠AI赋能机器人系统的基础性工作，本文提出一个面向机器人操作的公开工业基准。该基准基于NVIDIA Omniverse Isaac Sim仿真平台，涵盖八项代表性操作任务及多种AI软件控制器。我们通过大规模实验评估分析了AI控制器在解决机器人操作任务中的性能表现，为深入理解其有效性提供了依据。为进一步验证基准的适用性，我们开发了一个兼容物理仿真器与OpenAI Gym环境的反证框架，该框架弥合了传统测试方法与基于现代物理引擎仿真之间的鸿沟。通过反证测试，我们检验了不同优化方法在物理仿真环境下对AI赋能的机器人操作进行验证的有效性。本研究不仅为AI赋能机器人系统的设计与开发奠定基础，更为该领域从业者提供实践经验与指导，推动这一关键学术与工业领域的深入研究。