Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed. In this paper, we introduce Modular Transformers, a modularized encoder-decoder framework for flexible sequence-to-sequence model compression. Modular Transformers train modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation. After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs. Experimental results show that after a single training phase, by simply varying the assembling strategy, Modular Transformers can achieve flexible compression ratios from 1.1x to 6x with little to moderate relative performance drop.

翻译：预训练Transformer模型（如T5和BART）在广泛的文本生成任务中推动了前沿技术的发展。将这些模型压缩为更小的版本对于实际应用至关重要。常见的神经网络压缩技术（如知识蒸馏或量化）仅限于压缩比固定的静态压缩。本文提出模块化Transformer（Modular Transformers），一种用于灵活序列到序列模型压缩的模块化编码器-解码器框架。模块化Transformer通过模块替换和知识蒸馏训练模块化层，使其具备原始模型中两个或多个连续层的相同功能。训练完成后，模块化层可灵活组装成满足不同性能-效率权衡的序列到序列模型。实验结果表明，在单次训练阶段后，仅需改变组装策略，模块化Transformer即可实现从1.1倍到6倍的灵活压缩比，且性能下降幅度较小至中等。