Training models with varying capacities can be advantageous for deploying them in different scenarios. While high-capacity models offer better performance, low-capacity models require fewer computing resources for training and inference. In this work, we propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. This framework consists of two composite model architectures and a joint training algorithm called Two-Stage Joint-Training (TSJT). Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously, leading to faster and more efficient convergence. Extensive experiments on the multilingual machine translation benchmark WMT10 show that our method outperforms low-capacity baseline models and achieves comparable or better performance on high-capacity models. Notably, the analysis demonstrates that our method significantly influences the initial training process, leading to more efficient convergence and superior solutions.

翻译：针对不同部署场景，训练不同容量的模型具有实际优势：高容量模型性能更优，而低容量模型在训练和推理阶段所需计算资源更少。本文提出一种新颖的一站式训练框架，实现高容量与低容量模型的联合训练。该框架包含两种复合模型架构，以及名为两阶段联合训练（TSJT）的联合训练算法。与需要分别从头训练多容量模型的知识蒸馏方法不同，本方法通过同步整合不同容量模型的监督信号，实现更快速高效的收敛。在WMT10多语言机器翻译基准上的大量实验表明，本方法超越低容量基线模型，并在高容量模型上取得相当或更优性能。特别值得注意的是，分析结果表明本方法显著影响初始训练过程，从而达成更高效的收敛与更优的解。