Training state-of-the-art (SOTA) deep models often requires extensive data, resulting in substantial training and storage costs. To address these challenges, dataset condensation has been developed to learn a small synthetic set that preserves essential information from the original large-scale dataset. Nowadays, optimization-oriented methods have been the primary method in the field of dataset condensation for achieving SOTA results. However, the bi-level optimization process hinders the practical application of such methods to realistic and larger datasets. To enhance condensation efficiency, previous works proposed Distribution-Matching (DM) as an alternative, which significantly reduces the condensation cost. Nonetheless, current DM-based methods still yield less comparable results to SOTA optimization-oriented methods. In this paper, we argue that existing DM-based methods overlook the higher-order alignment of the distributions, which may lead to sub-optimal matching results. Inspired by this, we present a novel DM-based method named M3D for dataset condensation by Minimizing the Maximum Mean Discrepancy between feature representations of the synthetic and real images. By embedding their distributions in a reproducing kernel Hilbert space, we align all orders of moments of the distributions of real and synthetic images, resulting in a more generalized condensed set. Notably, our method even surpasses the SOTA optimization-oriented method IDC on the high-resolution ImageNet dataset. Extensive analysis is conducted to verify the effectiveness of the proposed method. Source codes are available at https://github.com/Hansong-Zhang/M3D.

翻译：训练最先进的深度模型通常需要大量数据，导致巨大的训练和存储成本。为应对这些挑战，数据集压缩技术被提出，旨在学习一个小型合成集，以保留原始大规模数据集中的关键信息。目前，优化导向方法已成为数据集压缩领域实现最优结果的主要手段。然而，双层优化过程阻碍了此类方法在实际且更大规模数据集上的应用。为提升压缩效率，先前研究提出了分布匹配方法作为替代方案，显著降低了压缩成本。但现有基于分布匹配的方法仍难以媲美优化导向方法的最优结果。本文认为，现有分布匹配方法忽视了分布的高阶对齐，可能导致次优的匹配结果。受此启发，我们提出了一种名为M3D的新型分布匹配方法，通过最小化合成图像和真实图像特征表示之间的最大均值差异来进行数据集压缩。通过将分布嵌入再生核希尔伯特空间，我们实现了真实与合成图像分布的所有阶矩对齐，从而获得更具泛化性的压缩集。值得注意的是，该方法在高分辨率ImageNet数据集上甚至超越了最先进的优化导向方法IDC。我们进行了广泛的分析以验证所提方法的有效性。源代码已开源至https://github.com/Hansong-Zhang/M3D。