We present a review of popular simulation engines and frameworks used in reinforcement learning (RL) research, aiming to guide researchers in selecting tools for creating simulated physical environments for RL and training setups. It evaluates nine frameworks (Brax, Chrono, Gazebo, MuJoCo, ODE, PhysX, PyBullet, Webots, and Unity) based on their popularity, feature range, quality, usability, and RL capabilities. We highlight the challenges in selecting and utilizing physics engines for RL research, including the need for detailed comparisons and an understanding of each framework's capabilities. Key findings indicate MuJoCo as the leading framework due to its performance and flexibility, despite usability challenges. Unity is noted for its ease of use but lacks scalability and simulation fidelity. The study calls for further development to improve simulation engines' usability and performance and stresses the importance of transparency and reproducibility in RL research. This review contributes to the RL community by offering insights into the selection process for simulation engines, facilitating informed decision-making.

翻译：本文综述了强化学习研究中常用的仿真引擎与框架，旨在为研究人员选择工具以创建强化学习的模拟物理环境和训练设置提供指导。文章基于流行度、功能范围、质量、易用性及强化学习能力，对九个框架（Brax、Chrono、Gazebo、MuJoCo、ODE、PhysX、PyBullet、Webots和Unity）进行了评估。我们重点探讨了为强化学习研究选择和利用物理引擎所面临的挑战，包括需要进行详细比较以及理解各框架的能力。主要发现表明，尽管存在易用性方面的挑战，MuJoCo因其性能和灵活性而成为领先的框架。Unity因其易用性而受到关注，但在可扩展性和仿真保真度方面有所欠缺。本研究呼吁进一步开发以提高仿真引擎的易用性和性能，并强调了强化学习研究中透明度和可重复性的重要性。本综述通过为仿真引擎的选择过程提供见解，促进明智的决策制定，从而为强化学习社区做出贡献。