Dataset distillation is the technique of synthesizing smaller condensed datasets from large original datasets while retaining necessary information to persist the effect. In this paper, we approach the dataset distillation problem from a novel perspective: we regard minimizing the prediction discrepancy on the real data distribution between models, which are respectively trained on the large original dataset and on the small distilled dataset, as a conduit for condensing information from the raw data into the distilled version. An adversarial framework is proposed to solve the problem efficiently. In contrast to existing distillation methods involving nested optimization or long-range gradient unrolling, our approach hinges on single-level optimization. This ensures the memory efficiency of our method and provides a flexible tradeoff between time and memory budgets, allowing us to distil ImageNet-1K using a minimum of only 6.5GB of GPU memory. Under the optimal tradeoff strategy, it requires only 2.5$\times$ less memory and 5$\times$ less runtime compared to the state-of-the-art. Empirically, our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets including ImageNet-1K, significantly surpassing state-of-the-art. Additionally, extensive tests reveal that our distilled datasets excel in cross-architecture generalization capabilities.

翻译：数据集蒸馏是从大规模原始数据集中合成更小规模压缩数据集的技术，同时保留必要信息以保持原始效果。本文从全新视角处理数据集蒸馏问题：我们将分别在大规模原始数据集和小规模蒸馏数据集上训练的模型，在真实数据分布上的预测差异最小化，视为将原始数据信息压缩至蒸馏版本的有效途径。我们提出对抗框架以高效解决该问题。与涉及嵌套优化或长程梯度展开的现有蒸馏方法不同，本方法基于单层优化，确保了内存效率，并能在时间与内存预算之间灵活权衡，从而仅需最低6.5GB GPU内存即可完成ImageNet-1K的蒸馏。在最优权衡策略下，相比于当前最优方法，本方法仅需其2.5倍以下的内存和5倍以下的运行时间。实验表明，本方法可生成仅为原始数据集大小10%的合成数据集，但在包括ImageNet-1K在内的完整原始数据集上训练模型的测试准确率平均达到94%，显著超越当前最优水平。此外，广泛测试表明，本方法蒸馏的数据集在跨架构泛化能力方面表现优异。