Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite recent progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.

翻译：数据集蒸馏旨在将整个数据集的知识浓缩到少量合成图像中，其核心思想是生成少量合成数据点，当这些数据点作为训练数据输入学习算法时，所得到的模型能够近似于基于原始数据训练的模型。尽管该领域近期取得进展，但现有数据集蒸馏方法在泛化到新架构以及扩展到高分辨率数据集方面仍存在不足。为解决上述问题，我们提出利用预训练深度生成模型中的学习先验来合成蒸馏数据。为实现这一目标，我们设计了一种新的优化算法，将大量图像蒸馏为生成模型潜空间中的少量中间特征向量。我们的方法对现有技术进行了增强，显著提升了所有设置下跨架构泛化的能力。