Deep learning has grown tremendously over recent years, yielding state-of-the-art results in various fields. However, training such models requires huge amounts of data, increasing the computational time and cost. To address this, dataset distillation was proposed to compress a large training dataset into a smaller synthetic one that retains its performance -- this is usually done by (1) uniformly initializing a synthetic set and (2) iteratively updating/learning this set according to a predefined loss by uniformly sampling instances from the full data. In this paper, we improve both phases of dataset distillation: (1) we present a provable, sampling-based approach for initializing the distilled set by identifying important and removing redundant points in the data, and (2) we further merge the idea of data subset selection with dataset distillation, by training the distilled set on ``important'' sampled points during the training procedure instead of randomly sampling the next batch. To do so, we define the notion of importance based on the relative contribution of instances with respect to two different loss functions, i.e., one for the initialization phase (a kernel fitting function for kernel ridge regression and $K$-means based loss function for any other distillation method), and the relative cross-entropy loss (or any other predefined loss) function for the training phase. Finally, we provide experimental results showing how our method can latch on to existing dataset distillation techniques and improve their performance.

翻译：近年来，深度学习取得了巨大发展，在各个领域都实现了最先进的成果。然而，训练此类模型需要大量数据，从而增加了计算时间和成本。为解决这一问题，提出了数据集蒸馏方法，将大型训练数据集压缩为较小的合成数据集，同时保持其性能——这通常通过以下两步完成：(1) 均匀初始化一个合成集，以及 (2) 通过从完整数据中均匀采样实例，根据预定义损失迭代更新/学习该集合。在本文中，我们改进了数据集蒸馏的两个阶段：(1) 提出了一种可证明的、基于采样的方法，通过识别重要数据点并去除冗余点来初始化蒸馏集；以及 (2) 进一步将数据子集选择的思想与数据集蒸馏相结合，在训练过程中基于“重要”采样点训练蒸馏集，而非随机采样下一批次。为此，我们根据实例在两个不同损失函数下的相对贡献来定义重要性的概念：一个用于初始化阶段（应用于核岭回归的核拟合函数，以及应用于任何其他蒸馏方法的基于 $K$-均值的损失函数），另一个用于训练阶段的相对交叉熵损失（或任何其他预定义损失）函数。最后，我们通过实验结果表明，我们的方法能够与现有数据集蒸馏技术相结合，并提升其性能。