Varying dynamics parameters in simulation is a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's behavior but notoriously leads to overly conservative policies when randomizing excessively. In this paper, we propose a novel approach to address sim-to-real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndomization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while retaining generalization capabilities. In achieving this, DORAEMON gradually increases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON in obtaining highly adaptive and generalizable policies, i.e. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. Notably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.

翻译：在仿真中变化动力学参数是一种流行的域随机化（DR）方法，用于克服强化学习（RL）中的现实差距。然而，DR高度依赖于动力学参数采样分布的选择，因为高变异性对于规范智能体行为至关重要，但过度随机化必然导致过于保守的策略。本文提出了一种新颖的模拟到现实迁移方法，该方法在仿真训练期间自动调整动力学分布，无需真实世界数据。我们提出了通过熵最大化实现域随机化（DORAEMON），这是一个约束优化问题，在保持泛化能力的同时直接最大化训练分布的熵。为实现这一目标，DORAEMON在保证当前策略成功率足够高的情况下，逐步增加采样动力学参数的多样性。我们通过实验验证了DORAEMON在获得高度适应性和泛化性策略方面的一致优势，即在最广泛的动力学参数范围内解决当前任务，优于DR文献中的代表性基线方法。值得注意的是，我们还在未知真实世界参数的机器人操作设置中，通过成功的零样本迁移证明了DORAEMON的模拟到现实应用可行性。