Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our experiments indicate that CLAFusion, with an extra finetuning process, improves the accuracy of residual networks on the CIFAR10, CIFAR100, and Tiny-ImageNet datasets. Furthermore, we explore its practical usage for model compression and knowledge distillation when applying to the teacher-student setting.

翻译：基于最优传输的逐层模型融合方法OTFusion（最优传输融合）通过软神经元关联统一不同预训练网络，以节省计算资源。尽管该方法取得了成功，但其要求输入网络具有相同的层数。为解决此问题，我们提出一种新型模型融合框架CLAFusion（跨层对齐融合），通过跨层对齐实现不同层数的神经网络（即异构神经网络）融合。跨层对齐问题本质上是一个非平衡分配问题，可通过动态规划高效求解。基于该对齐结果，本框架在逐层模型融合前先均衡神经网络层数。实验表明，结合额外微调过程的CLAFusion能提升残差网络在CIFAR10、CIFAR100及Tiny-ImageNet数据集上的准确率。进一步地，我们探索了该方法在教师-学生场景下应用于模型压缩与知识蒸馏的实用价值。