Varying dynamics parameters in simulation is a popular Domain Randomization (DR) approach for overcoming the reality gap in Reinforcement Learning (RL). Nevertheless, DR heavily hinges on the choice of the sampling distribution of the dynamics parameters, since high variability is crucial to regularize the agent's behavior but notoriously leads to overly conservative policies when randomizing excessively. In this paper, we propose a novel approach to address sim-to-real transfer, which automatically shapes dynamics distributions during training in simulation without requiring real-world data. We introduce DOmain RAndomization via Entropy MaximizatiON (DORAEMON), a constrained optimization problem that directly maximizes the entropy of the training distribution while retaining generalization capabilities. In achieving this, DORAEMON gradually increases the diversity of sampled dynamics parameters as long as the probability of success of the current policy is sufficiently high. We empirically validate the consistent benefits of DORAEMON in obtaining highly adaptive and generalizable policies, i.e. solving the task at hand across the widest range of dynamics parameters, as opposed to representative baselines from the DR literature. Notably, we also demonstrate the Sim2Real applicability of DORAEMON through its successful zero-shot transfer in a robotic manipulation setup under unknown real-world parameters.

翻译：在仿真中变化动力学参数是一种流行的域随机化方法，用于克服强化学习中的现实差距。然而，域随机化严重依赖于动力学参数采样分布的选择，因为高变异性对于正则化智能体行为至关重要，但过度随机化时通常会导致过于保守的策略。本文提出了一种解决仿真到现实迁移问题的新方法，该方法在仿真训练期间自动调整动力学分布，无需真实世界数据。我们引入了通过熵最大化实现的域随机化（DORAEMON），这是一个约束优化问题，直接最大化训练分布的熵，同时保持泛化能力。通过实现这一点，DORAEMON在成功概率足够高时逐步增加采样动力学参数的多样性。我们通过实验验证了DORAEMON在获得高度适应性和泛化性策略方面的一致优势，即与域随机化文献中的代表性基线相比，能够在最广泛的动力学参数范围内解决当前任务。值得注意的是，我们还通过在未知真实世界参数的机器人操控设置中成功实现零样本迁移，展示了DORAEMON在仿真到现实应用中的可行性。