Imitation learning uses data for training policies to solve complex tasks. However, when the training data is collected from human demonstrators, it often leads to multimodal distributions because of the variability in human actions. Most imitation learning methods rely on a maximum likelihood (ML) objective to learn a parameterized policy, but this can result in suboptimal or unsafe behavior due to the mode-averaging property of the ML objective. In this work, we propose Information Maximizing Curriculum, a curriculum-based approach that assigns a weight to each data point and encourages the model to specialize in the data it can represent, effectively mitigating the mode-averaging problem by allowing the model to ignore data from modes it cannot represent. To cover all modes and thus, enable diverse behavior, we extend our approach to a mixture of experts (MoE) policy, where each mixture component selects its own subset of the training data for learning. A novel, maximum entropy-based objective is proposed to achieve full coverage of the dataset, thereby enabling the policy to encompass all modes within the data distribution. We demonstrate the effectiveness of our approach on complex simulated control tasks using diverse human demonstrations, achieving superior performance compared to state-of-the-art methods.

翻译：模仿学习利用数据训练策略以解决复杂任务。然而，当训练数据来自人类示范者时，由于人类行为的变异性，通常会导致多模态分布。大多数模仿学习方法依赖最大似然目标来学习参数化策略，但最大似然目标的模式平均特性可能导致次优或不安全的行为。本文提出信息最大化课程，一种基于课程学习的方法，通过为每个数据点分配权重，鼓励模型专注于其能表示的数据，有效缓解模式平均问题——允许模型忽略其无法表示的模式对应的数据。为覆盖所有模式以实现多样化行为，我们将该方法扩展到混合专家策略，其中每个混合组件选择自身训练数据子集进行学习。我们提出一种基于最大熵的新型目标，以实现对数据集的完整覆盖，从而让策略能够囊括数据分布中的所有模式。在涉及多样化人类示范的复杂仿真控制任务中，我们验证了该方法的效果，相较于现有最先进方法取得了更优性能。