Simulation is essential to reinforcement learning (RL) before implementation in the real world, especially for safety-critical applications like robot manipulation. Conventionally, RL agents are sensitive to the discrepancies between the simulation and the real world, known as the sim-to-real gap. The application of domain randomization, a technique used to fill this gap, is limited to the imposition of heuristic-randomized models. We investigate the properties of intrinsic stochasticity of real-time simulation (RT-IS) of off-the-shelf simulation software and its potential to improve the robustness of RL methods and the performance of domain randomization. Firstly, we conduct analytical studies to measure the correlation of RT-IS with the occupation of the computer hardware and validate its comparability with the natural stochasticity of a physical robot. Then, we apply the RT-IS feature in the training of an RL agent. The simulation and physical experiment results verify the feasibility and applicability of RT-IS to robust RL agent design for robot manipulation tasks. The RT-IS-powered robust RL agent outperforms conventional RL agents on robots with modeling uncertainties. It requires fewer heuristic randomization and achieves better generalizability than the conventional domain-randomization-powered agents. Our findings provide a new perspective on the sim-to-real problem in practical applications like robot manipulation tasks.

翻译：强化学习在实际部署到现实世界之前，仿真至关重要，尤其是在机器人操作等安全关键型应用中。传统上，强化学习智能体对仿真与真实世界之间的差异（即仿真到现实差距）非常敏感。领域随机化技术虽被用于弥合这一差距，但其应用局限于施加启发式随机化模型。我们研究了现成仿真软件中实时仿真内在随机性（RT-IS）的特性，及其在提升强化学习方法鲁棒性和领域随机化性能方面的潜力。首先，我们开展分析性研究，测量RT-IS与计算机硬件占用的相关性，并验证其与物理机器人自然随机性的可比性。然后，我们将RT-IS特征应用于强化学习智能体的训练。仿真与物理实验结果验证了RT-IS在机器人操作任务中设计鲁棒强化学习智能体的可行性和适用性。与传统的强化学习智能体相比，基于RT-IS的鲁棒智能体在存在建模不确定性的机器人上表现更优；与传统的基于领域随机化的智能体相比，它需要的启发式随机化更少，且泛化能力更强。我们的发现为机器人操作等实际应用中的仿真到现实问题提供了新视角。