We need billion-scale images to achieve more generalizable and ground-breaking vision models, as well as massive dataset storage to ship the images (e.g., the LAION-4B dataset needs 240TB storage space). However, it has become challenging to deal with unlimited dataset storage with limited storage infrastructure. A number of storage-efficient training methods have been proposed to tackle the problem, but they are rarely scalable or suffer from severe damage to performance. In this paper, we propose a storage-efficient training strategy for vision classifiers for large-scale datasets (e.g., ImageNet) that only uses 1024 tokens per instance without using the raw level pixels; our token storage only needs <1% of the original JPEG-compressed raw pixels. We also propose token augmentations and a Stem-adaptor module to make our approach able to use the same architecture as pixel-based approaches with only minimal modifications on the stem layer and the carefully tuned optimization settings. Our experimental results on ImageNet-1k show that our method significantly outperforms other storage-efficient training methods with a large gap. We further show the effectiveness of our method in other practical scenarios, storage-efficient pre-training, and continual learning. Code is available at https://github.com/naver-ai/seit

翻译：我们需要数十亿规模的图像来实现更具泛化能力和突破性的视觉模型，同时需要海量数据集存储来承载这些图像（例如，LAION-4B数据集需要240TB存储空间）。然而，在有限的存储基础设施下处理无限数据集存储已成为一项挑战。为了解决这一问题，已有多种节能存储训练方法被提出，但它们要么可扩展性不足，要么会严重损害模型性能。本文提出了一种面向大规模数据集（如ImageNet）视觉分类器的节能存储训练策略，该方法仅使用每个实例1024个令牌且无需原始像素；我们的令牌存储仅需原始JPEG压缩像素的不足1%。我们还提出了令牌增强方法和Stem-Adaptor模块，使得我们的方法能够在仅对主干层和精心调优的优化设置进行最小修改的前提下，使用与基于像素的方法相同的架构。我们在ImageNet-1k上的实验结果表明，本方法以显著优势优于其他节能存储训练方法。我们进一步展示了该方法在其他实际场景（如节能存储预训练和持续学习）中的有效性。代码已开源在 https://github.com/naver-ai/seit