Dataset Distillation aims to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. Despite recent progress in the field, existing dataset distillation methods fail to generalize to new architectures and scale to high-resolution datasets. To overcome the above issues, we propose to use the learned prior from pre-trained deep generative models to synthesize the distilled data. To achieve this, we present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space. Our method augments existing techniques, significantly improving cross-architecture generalization in all settings.

翻译：数据集蒸馏旨在将整个数据集的知识浓缩到少量合成图像中。其核心思想是合成少量数据点，当这些数据点作为训练数据提供给学习算法时，能够训练出近似于在原始数据上训练得到的模型。尽管该领域近期取得了进展，但现有数据集蒸馏方法难以泛化至新架构，也无法扩展到高分辨率数据集。为克服上述问题，我们提出利用预训练深度生成模型的学习先验来合成蒸馏数据。为此，我们提出一种新型优化算法，可将大量图像蒸馏为生成模型潜在空间中的少量中间特征向量。本方法对现有技术进行增强，显著提升了所有场景下的跨架构泛化性能。