Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

翻译：广泛观察到的神经缩放定律，即误差随训练集规模、模型规模或两者兼有的幂次下降，推动了深度学习性能的显著提升。然而，仅通过缩放实现的这些改进需要巨大的计算和能源成本。本文聚焦于误差随数据集规模的缩放关系，并从理论上证明：如果我们拥有高质量的数据剪枝指标，能够对训练样本的舍弃顺序进行排序以实现任意剪枝后的数据集规模，那么我们可以突破幂律缩放的限制，甚至可能将其降低为指数缩放。随后，我们通过实验验证了这种剪枝数据集规模下的改进缩放预测，在CIFAR-10、SVHN和ImageNet上训练的ResNet中，实际观察到了优于幂律缩放的现象。此外，鉴于寻找高质量剪枝指标的重要性，我们首次在ImageNet上对十种不同的数据剪枝指标进行了大规模基准研究。我们发现，大多数现有高性能指标难以扩展到ImageNet，而表现最佳的指标计算强度大，且需要为每张图像提供标签。因此，我们开发了一种新颖、简单、廉价且可扩展的自监督剪枝指标，其性能与最佳监督指标相当。总体而言，我们的研究表明，发现良好的数据剪枝指标可能为大幅改进神经缩放定律提供可行路径，从而降低现代深度学习的资源成本。