Training a robust policy is critical for policy deployment in real-world systems or dealing with unknown dynamics mismatch in different dynamic systems. Domain Randomization~(DR) is a simple and elegant approach that trains a conservative policy to counter different dynamic systems without expert knowledge about the target system parameters. However, existing works reveal that the policy trained through DR tends to be over-conservative and performs poorly in target domains. Our key insight is that dynamic systems with different parameters provide different levels of difficulty for the policy, and the difficulty of behaving well in a system is constantly changing due to the evolution of the policy. If we can actively sample the systems with proper difficulty for the policy on the fly, it will stabilize the training process and prevent the policy from becoming over-conservative or over-optimistic. To operationalize this idea, we introduce Active Dynamics Preference~(ADP), which quantifies the informativeness and density of sampled system parameters. ADP actively selects system parameters with high informativeness and low density. We validate our approach in four robotic locomotion tasks with various discrepancies between the training and testing environments. Extensive results demonstrate that our approach has superior robustness for system inconsistency compared to several baselines.

翻译：为在现实系统中部署策略或应对不同动力学系统间的未知失配问题，训练鲁棒策略至关重要。域随机化方法无需目标系统参数的专业知识，通过训练保守策略来应对不同动力学系统，是一种简洁优雅的途径。然而，现有研究表明，经域随机化训练的策略往往过于保守，在目标域中表现欠佳。我们的核心见解在于：不同参数的动力系统为策略提供了不同难度水平，且由于策略的演化，在系统中表现良好的难度也在持续变化。若能根据策略实时主动采样适当难度的系统，将有助于稳定训练过程，防止策略变得过于保守或过于乐观。为实现这一思想，我们提出主动动力学偏好方法，该方法量化了采样系统参数的信息量和密度。主动动力学偏好主动选择具有高信息量和低密度的系统参数。我们在四个机器人运动任务中验证了该方法，这些任务在训练环境与测试环境之间存在多种差异。大量实验结果表明，与多种基线方法相比，本方法对系统不一致性具有更优越的鲁棒性。