The superior performance of modern visual backbones usually comes with a costly training procedure. We contribute to this issue by generalizing the idea of curriculum learning beyond its original formulation, i.e., training models using easier-to-harder data. Specifically, we reformulate the training curriculum as a soft-selection function, which uncovers progressively more difficult patterns within each example during training, instead of performing easier-to-harder sample selection. Our work is inspired by an intriguing observation on the learning dynamics of visual backbones: during the earlier stages of training, the model predominantly learns to recognize some 'easier-to-learn' discriminative patterns in the data. These patterns, when observed through frequency and spatial domains, incorporate lower-frequency components, and the natural image contents without distortion or data augmentation. Motivated by these findings, we propose a curriculum where the model always leverages all the training data at every learning stage, yet the exposure to the 'easier-to-learn' patterns of each example is initiated first, with harder patterns gradually introduced as training progresses. To implement this idea in a computationally efficient way, we introduce a cropping operation in the Fourier spectrum of the inputs, enabling the model to learn from only the lower-frequency components. Then we show that exposing the contents of natural images can be readily achieved by modulating the intensity of data augmentation. Finally, we integrate these aspects and design curriculum schedules with tailored search algorithms. The resulting method, EfficientTrain++, is simple, general, yet surprisingly effective. It reduces the training time of a wide variety of popular models by 1.5-3.0x on ImageNet-1K/22K without sacrificing accuracy. It also demonstrates efficacy in self-supervised learning (e.g., MAE).

翻译：现代视觉骨干网络的卓越性能通常伴随着高昂的训练成本。我们通过将课程学习思想推广至其原始表述之外（即使用由易到难的数据训练模型）来应对这一问题。具体而言，我们将训练课程重构为一种软选择函数，该函数在训练过程中逐步揭示每个样本中越来越困难的模式，而非执行由易到难的样本选择。我们的工作源于对视觉骨干网络学习动态的一个有趣观察：在训练早期阶段，模型主要学习识别数据中某些"更易学习"的判别模式。这些模式在频率域和空间域中体现为低频分量、未失真或未经过数据增强的自然图像内容。受此启发，我们提出一种课程学习方案：模型在每个学习阶段始终利用全部训练数据，但首先暴露每个样本的"更易学习"模式，随着训练推进逐渐引入更困难的模式。为以计算高效的方式实现该思想，我们在输入信号的傅里叶频谱中引入裁剪操作，使模型仅从低频分量中学习。进而证明通过调节数据增强强度即可有效暴露自然图像内容。最后我们整合这些要素，设计具有定制化搜索算法的课程调度策略。由此产生的EfficientTrain++方法简洁通用，却出人意料地有效：在ImageNet-1K/22K上，它能将多种流行模型的训练时间缩短1.5-3.0倍且不损失精度，同时在自监督学习（如MAE）中也展现出有效性。