Recent years have witnessed a remarkable success of large deep learning models. However, training these models is challenging due to high computational costs, painfully slow convergence, and overfitting issues. In this paper, we present Deep Incubation, a novel approach that enables the efficient and effective training of large models by dividing them into smaller sub-modules that can be trained separately and assembled seamlessly. A key challenge for implementing this idea is to ensure the compatibility of the independently trained sub-modules. To address this issue, we first introduce a global, shared meta model, which is leveraged to implicitly link all the modules together, and can be designed as an extremely small network with negligible computational overhead. Then we propose a module incubation algorithm, which trains each sub-module to replace the corresponding component of the meta model and accomplish a given learning task. Despite the simplicity, our approach effectively encourages each sub-module to be aware of its role in the target large model, such that the finally-learned sub-modules can collaborate with each other smoothly after being assembled. Empirically, our method outperforms end-to-end (E2E) training in terms of both final accuracy and training efficiency. For example, on top of ViT-Huge, it improves the accuracy by 2.7% on ImageNet or achieves similar performance with 4x less training time. Notably, the gains are significant for downstream tasks as well (e.g., object detection and image segmentation on COCO and ADE20K). Code is available at https://github.com/LeapLabTHU/Deep-Incubation.

翻译：近年来，大型深度学习模型取得了显著成功。然而，训练这些模型面临巨大挑战，包括高昂的计算成本、极其缓慢的收敛速度以及过拟合问题。本文提出深度孵化方法，这是一种将大型模型分解为可独立训练并能无缝组装的小型子模块，从而实现高效训练的新范式。该方案的关键挑战在于确保独立训练的子模块之间具有兼容性。为解决此问题，我们首先引入一个全局共享的元模型，该模型可隐式连接所有子模块，且可设计为计算开销极小的超小型网络。然后提出模块孵化算法，通过训练每个子模块替代元模型中对应组件，完成给定学习任务。尽管方法简洁，但能有效促使每个子模块感知其在目标大型模型中的角色，使得最终习得的子模块在组装后能够顺畅协作。实验表明，我们的方法在最终精度和训练效率上均优于端到端训练。例如，基于ViT-Huge模型，该方法在ImageNet上提升2.7%的准确率，或在4倍更少的训练时间内达到相近性能。值得注意的是，该方法在下游任务（如COCO数据集上的目标检测和ADE20K数据集上的图像分割）中同样取得显著提升。代码已开源至https://github.com/LeapLabTHU/Deep-Incubation。