Most dataset distillation methods struggle to accommodate large-scale datasets due to their substantial computational and memory requirements. In this paper, we present a curriculum-based dataset distillation framework designed to harmonize scalability with efficiency. This framework strategically distills synthetic images, adhering to a curriculum that transitions from simple to complex. By incorporating curriculum evaluation, we address the issue of previous methods generating images that tend to be homogeneous and simplistic, doing so at a manageable computational cost. Furthermore, we introduce adversarial optimization towards synthetic images to further improve their representativeness and safeguard against their overfitting to the neural network involved in distilling. This enhances the generalization capability of the distilled images across various neural network architectures and also increases their robustness to noise. Extensive experiments demonstrate that our framework sets new benchmarks in large-scale dataset distillation, achieving substantial improvements of 11.1\% on Tiny-ImageNet, 9.0\% on ImageNet-1K, and 7.3\% on ImageNet-21K. The source code will be released to the community.

翻译：大多数数据集蒸馏方法由于需要大量的计算资源和内存，难以适应大规模数据集。本文提出了一种基于课程学习的数据集蒸馏框架，旨在兼顾可扩展性与效率。该框架按照从简单到复杂的课程顺序策略性地蒸馏合成图像。通过引入课程评估，我们解决了先前方法生成图像趋于同质化和单一化的问题，同时将计算成本控制在可管理范围内。此外，我们针对合成图像引入了对抗优化，以进一步提升其代表性，并防止其对参与蒸馏的神经网络产生过拟合。这增强了蒸馏图像在不同神经网络架构间的泛化能力，并提高了其对噪声的鲁棒性。大量实验表明，我们的框架在大规模数据集蒸馏上树立了新的标杆，在Tiny-ImageNet、ImageNet-1K和ImageNet-21K上分别取得了11.1%、9.0%和7.3%的显著提升。源代码将向社区公开。