State-of-the-art deep neural networks are trained with large amounts (millions or even billions) of data. The expensive computation and memory costs make it difficult to train them on limited hardware resources, especially for recent popular large language models (LLM) and computer vision models (CV). Recent popular dataset distillation methods are thus developed, aiming to reduce the number of training samples via synthesizing small-scale datasets via gradient matching. However, as the gradient calculation is coupled with the specific network architecture, the synthesized dataset is biased and performs poorly when used for training unseen architectures. To address these limitations, we present dataset quantization (DQ), a new framework to compress large-scale datasets into small subsets which can be used for training any neural network architectures. Extensive experiments demonstrate that DQ is able to generate condensed small datasets for training unseen network architectures with state-of-the-art compression ratios for lossless model training. To the best of our knowledge, DQ is the first method that can successfully distill large-scale datasets such as ImageNet-1k with a state-of-the-art compression ratio. Notably, with 60% data from ImageNet and 20% data from Alpaca's instruction tuning data, the models can be trained with negligible or no performance drop for both vision tasks (including classification, semantic segmentation, and object detection) as well as language tasks (including instruction tuning tasks such as BBH and DROP).

翻译：最先进的深度神经网络需要大量数据（百万甚至数十亿级别）进行训练，其高昂的计算和内存成本使其在有限硬件资源上难以训练，尤其是近期流行的大型语言模型（LLM）和计算机视觉模型（CV）。为此，近期发展了流行的数据集蒸馏方法，旨在通过梯度匹配合成小规模数据集来减少训练样本数量。然而，由于梯度计算与特定网络架构耦合，合成数据集存在偏差，在训练未见过的架构时表现不佳。为解决这些局限性，我们提出数据集量化（DQ）这一新框架，将大规模数据集压缩成可用于训练任意神经网络架构的小子集。大量实验表明，DQ能够生成用于训练未见过的网络架构的浓缩小数据集，并在无损模型训练中实现最先进的压缩比。据我们所知，DQ是首个成功蒸馏ImageNet-1k等大规模数据集且达到最先进压缩比的方法。值得注意的是，使用ImageNet的60%数据和Alpaca指令微调数据的20%，模型在视觉任务（包括分类、语义分割和目标检测）以及语言任务（包括BBH和DROP等指令微调任务）上均能实现可忽略甚至零性能损失。