The superior performance of modern deep networks usually comes with a costly training procedure. This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers). Our work is inspired by the inherent learning dynamics of deep networks: we experimentally show that at an earlier training stage, the model mainly learns to recognize some 'easier-to-learn' discriminative patterns within each example, e.g., the lower-frequency components of images and the original information before data augmentation. Driven by this phenomenon, we propose a curriculum where the model always leverages all the training data at each epoch, while the curriculum starts with only exposing the 'easier-to-learn' patterns of each example, and introduces gradually more difficult patterns. To implement this idea, we 1) introduce a cropping operation in the Fourier spectrum of the inputs, which enables the model to learn from only the lower-frequency components efficiently, 2) demonstrate that exposing the features of original images amounts to adopting weaker data augmentation, and 3) integrate 1) and 2) and design a curriculum learning schedule with a greedy-search algorithm. The resulting approach, EfficientTrain, is simple, general, yet surprisingly effective. In the absence of hyper-parameter tuning, it reduces the training wall-time of a wide variety of popular models (e.g., ResNet, ConvNeXt, DeiT, PVT, Swin, and CSWin) by >1.5x on ImageNet-1K/22K without sacrificing the accuracy. It is also effective for self-supervised learning (e.g., MAE). Code is available at https://github.com/LeapLabTHU/EfficientTrain.

翻译：现代深度网络的卓越性能通常伴随着高昂的训练成本。本文提出了一种新的课程学习方法，用于高效训练视觉骨干网络（如视觉Transformer）。我们的工作受深度网络内在学习动态的启发：通过实验发现，在训练早期阶段，模型主要学习识别每个样本中"更易学习"的判别性模式，例如图像的低频分量以及数据增强前的原始信息。基于此现象，我们设计了一种课程方案：模型在每个训练周期中始终使用全部训练数据，但课程开始时仅暴露每个样本中"更易学习"的模式，并逐步引入更困难的模式。为实现这一想法，我们（1）在输入的傅里叶频谱中引入裁剪操作，使模型能够高效地从低频分量中学习；（2）证明暴露原始图像特征等同于采用较弱的数据增强；（3）整合（1）和（2），并通过贪心搜索算法设计课程学习计划。所提出的方法EfficientTrain简单、通用且效果显著。在无需超参数调优的情况下，它将ImageNet-1K/22K上多种主流模型（如ResNet、ConvNeXt、DeiT、PVT、Swin和CSWin）的训练耗时减少>1.5倍，同时保持精度不损失。该方法同样适用于自监督学习（如MAE）。代码已开源：https://github.com/LeapLabTHU/EfficientTrain。