Although multi-task deep neural network (DNN) models have computation and storage benefits over individual single-task DNN models, they can be further optimized via model compression. Numerous structured pruning methods are already developed that can readily achieve speedups in single-task models, but the pruning of multi-task networks has not yet been extensively studied. In this work, we investigate the effectiveness of structured pruning on multi-task models. We use an existing single-task filter pruning criterion and also introduce an MTL-based filter pruning criterion for estimating the filter importance scores. We prune the model using an iterative pruning strategy with both pruning methods. We show that, with careful hyper-parameter tuning, architectures obtained from different pruning methods do not have significant differences in their performances across tasks when the number of parameters is similar. We also show that iterative structure pruning may not be the best way to achieve a well-performing pruned model because, at extreme pruning levels, there is a high drop in performance across all tasks. But when the same models are randomly initialized and re-trained, they show better results.

翻译：尽管多任务深度神经网络模型相比单任务深度神经网络模型在计算和存储方面具有优势，但仍可通过模型压缩进一步优化。目前已开发出多种结构化剪枝方法可有效加速单任务模型，但多任务网络的剪枝问题尚未得到广泛研究。本研究探讨了结构化剪枝在多任务模型上的有效性。我们采用现有的单任务滤波器剪枝准则，并引入一种基于多任务学习的滤波器剪枝准则来估计滤波器重要性分数。通过迭代剪枝策略结合两种剪枝方法对模型进行剪枝。研究表明，在精细调节超参数的情况下，不同剪枝方法获得的架构在参数数量相近时，其在各项任务上的性能并无显著差异。同时我们指出，迭代式结构剪枝可能并非获得性能优良剪枝模型的最佳方案，因为在极端剪枝水平下，所有任务的性能均会出现大幅下降。但当相同模型经过随机初始化并重新训练后，其表现更为优异。