Conventional dataset distillation requires significant computational resources and assumes access to the entire dataset, an assumption impractical as it presumes all data resides on a central server. In this paper, we focus on dataset distillation in practical scenarios with access to only a fraction of the entire dataset. We introduce a novel distillation method that augments the conventional process by incorporating general model knowledge via the addition of Deep KKT (DKKT) loss. In practical settings, our approach showed improved performance compared to the baseline distribution matching distillation method on the CIFAR-10 dataset. Additionally, we present experimental evidence that Deep Support Vectors (DSVs) offer unique information to the original distillation, and their integration results in enhanced performance.

翻译：传统数据集蒸馏需要大量计算资源，并假设能够访问整个数据集，这一假设在实际中不切实际，因为它预设所有数据都位于中央服务器上。本文聚焦于仅能访问整个数据集一部分的实际场景中的数据集蒸馏问题。我们提出了一种新颖的蒸馏方法，通过引入深度KKT（DKKT）损失来融入通用模型知识，从而增强传统过程。在实际场景中，我们的方法在CIFAR-10数据集上相较于基线分布匹配蒸馏方法展现出更优性能。此外，我们通过实验证明，深度支持向量（DSV）为原始蒸馏提供了独特信息，其融合能进一步提升性能。