The efficacy of machine learning has traditionally relied on the availability of increasingly larger datasets. However, large datasets pose storage challenges and contain non-influential samples, which could be ignored during training without impacting the final accuracy of the model. In response to these limitations, the concept of distilling the information on a dataset into a condensed set of (synthetic) samples, namely a distilled dataset, emerged. One crucial aspect is the selected architecture (usually ConvNet) for linking the original and synthetic datasets. However, the final accuracy is lower if the employed model architecture differs from the model used during distillation. Another challenge is the generation of high-resolution images, e.g., 128x128 and higher. In this paper, we propose Latent Dataset Distillation with Diffusion Models (LD3M) that combine diffusion in latent space with dataset distillation to tackle both challenges. LD3M incorporates a novel diffusion process tailored for dataset distillation, which improves the gradient norms for learning synthetic images. By adjusting the number of diffusion steps, LD3M also offers a straightforward way of controlling the trade-off between speed and accuracy. We evaluate our approach in several ImageNet subsets and for high-resolution images (128x128 and 256x256). As a result, LD3M consistently outperforms state-of-the-art distillation techniques by up to 4.8 p.p. and 4.2 p.p. for 1 and 10 images per class, respectively.

翻译：机器学习的效果传统上依赖于可用数据集规模的不断增加。然而，大数据集带来存储挑战，且包含对训练无影响的样本，这些样本在训练过程中可被忽略而不影响模型最终精度。为应对这些局限，将数据集信息蒸馏为一组紧凑的（合成）样本（即蒸馏数据集）的概念应运而生。其中一个关键方面是选择用于连接原始数据集与合成数据集的特定架构（通常是卷积神经网络）。然而，若所采用的模型架构与蒸馏时使用的模型不同，最终精度会降低。另一挑战是生成高分辨率图像（例如128×128及以上）。本文提出基于扩散模型的潜在数据集蒸馏方法（LD3M），该方法将潜在空间扩散与数据集蒸馏相结合，以应对上述两大挑战。LD3M引入了一种专为数据集蒸馏设计的全新扩散过程，通过优化梯度范数来学习合成图像。通过调整扩散步数，LD3M还提供了一种控制速度与精度权衡的简便方法。我们在多个ImageNet子集及高分辨率图像（128×128和256×256）上评估了该方法。结果表明，在每个类别1张和10张图像的情况下，LD3M相比最先进的蒸馏技术分别持续提升高达4.8个百分点和4.2个百分点。