Dataset distillation reduces the network training cost by synthesizing small and informative datasets from large-scale ones. Despite the success of the recent dataset distillation algorithms, three drawbacks still limit their wider application: i). the synthetic images perform poorly on large architectures; ii). they need to be re-optimized when the distillation ratio changes; iii). the limited diversity restricts the performance when the distillation ratio is large. In this paper, we propose a novel distillation scheme to \textbf{D}istill information of large train sets \textbf{i}nto generative \textbf{M}odels, named DiM. Specifically, DiM learns to use a generative model to store the information of the target dataset. During the distillation phase, we minimize the differences in logits predicted by a models pool between real and generated images. At the deployment stage, the generative model synthesizes various training samples from random noises on the fly. Due to the simple yet effective designs, the trained DiM can be directly applied to different distillation ratios and large architectures without extra cost. We validate the proposed DiM across 4 datasets and achieve state-of-the-art results on all of them. To the best of our knowledge, we are the first to achieve higher accuracy on complex architectures than simple ones, such as 75.1\% with ResNet-18 and 72.6\% with ConvNet-3 on ten images per class of CIFAR-10. Besides, DiM outperforms previous methods with 10\% $\sim$ 22\% when images per class are 1 and 10 on the SVHN dataset.

翻译：数据集蒸馏通过从大规模数据集中合成小型且信息丰富的数据集来降低网络训练成本。尽管近期数据集蒸馏算法取得了成功，但三个缺陷仍限制了其更广泛的应用：i) 合成图像在大型架构上表现不佳；ii) 当蒸馏比例变化时需要重新优化；iii) 当蒸馏比例较大时，有限的多样性限制了性能。本文提出了一种新颖的蒸馏方案，将大型训练集的信息蒸馏到生成模型中，命名为DiM。具体而言，DiM学习使用生成模型存储目标数据集的信息。在蒸馏阶段，我们最小化真实图像与生成图像之间由模型池预测的对数几率差异。在部署阶段，生成模型从随机噪声中即时合成各种训练样本。得益于简单而有效的设计，训练好的DiM可直接适用于不同蒸馏比例和大型架构，而无需额外成本。我们在4个数据集上验证了所提出的DiM，并在所有数据集上达到了最优结果。据我们所知，我们是首个在复杂架构上取得比简单架构更高准确率的方法，例如在CIFAR-10每类十张图像上，ResNet-18达到75.1%，ConvNet-3达到72.6%。此外，在SVHN数据集上，当每类图像数为1和10时，DiM相比先前方法提升了10%至22%。