Dataset distillation aims to condense large datasets into a small number of synthetic examples that can be used as drop-in replacements when training new models. It has applications to interpretability, neural architecture search, privacy, and continual learning. Despite strong successes in supervised domains, such methods have not yet been extended to reinforcement learning, where the lack of a fixed dataset renders most distillation methods unusable. Filling the gap, we formalize behaviour distillation, a setting that aims to discover and then condense the information required for training an expert policy into a synthetic dataset of state-action pairs, without access to expert data. We then introduce Hallucinating Datasets with Evolution Strategies (HaDES), a method for behaviour distillation that can discover datasets of just four state-action pairs which, under supervised learning, train agents to competitive performance levels in continuous control tasks. We show that these datasets generalize out of distribution to training policies with a wide range of architectures and hyperparameters. We also demonstrate application to a downstream task, namely training multi-task agents in a zero-shot fashion. Beyond behaviour distillation, HaDES provides significant improvements in neuroevolution for RL over previous approaches and achieves SoTA results on one standard supervised dataset distillation task. Finally, we show that visualizing the synthetic datasets can provide human-interpretable task insights.

翻译：数据集蒸馏旨在将大规模数据集压缩为少量合成样本，这些样本可作为训练新模型时的直接替代品。该技术在可解释性、神经架构搜索、隐私保护和持续学习等领域具有应用价值。尽管在监督学习领域取得了显著成功，此类方法尚未扩展到强化学习领域——由于缺乏固定数据集，大多数蒸馏方法在此场景中无法适用。为填补这一空白，我们正式提出行为蒸馏这一设定，其目标是在无法访问专家数据的情况下，发现并压缩训练专家策略所需的信息，将其凝练为状态-动作对的合成数据集。我们进一步提出基于进化策略的数据集幻觉方法（HaDES），这是一种行为蒸馏方法，能够发现仅包含四组状态-动作对的数据集，通过监督学习即可在连续控制任务中训练出具有竞争力的智能体。实验表明这些数据集具备分布外泛化能力，可适用于不同架构和超参数配置的策略训练。我们还演示了该方法在下游任务中的应用，即以零样本方式训练多任务智能体。除行为蒸馏外，HaDES在强化学习的神经进化领域较先前方法取得显著改进，并在标准监督式数据集蒸馏任务中达到当前最优水平。最后，我们通过可视化合成数据集展示了其可提供人类可理解的任务洞察。