Model compression techniques reduce the computational load and memory consumption of deep neural networks. After the compression operation, e.g. parameter pruning, the model is normally fine-tuned on the original training dataset to recover from the performance drop caused by compression. However, the training data is not always available due to privacy issues or other factors. In this work, we present a data-free fine-tuning approach for pruning the backbone of deep neural networks. In particular, the pruned network backbone is trained with synthetically generated images, and our proposed intermediate supervision to mimic the unpruned backbone's output feature map. Afterwards, the pruned backbone can be combined with the original network head to make predictions. We generate synthetic images by back-propagating gradients to noise images while relying on L1-pruning for the backbone pruning. In our experiments, we show that our approach is task-independent due to pruning only the backbone. By evaluating our approach on 2D human pose estimation, object detection, and image classification, we demonstrate promising performance compared to the unpruned model. Our code is available at https://github.com/holzbock/dfbf.

翻译：模型压缩技术可降低深度神经网络的计算负载与内存消耗。在进行参数剪枝等压缩操作后，通常需要在原始训练数据集上对模型进行微调，以恢复压缩导致的性能下降。然而，由于隐私问题或其他因素，训练数据并非始终可用。本文提出了一种无需训练数据的微调方法，用于对深度神经网络的主干网络进行剪枝。具体而言，我们通过合成生成的图像训练剪枝后的主干网络，并采用本文提出的中间监督机制来模仿未剪枝主干网络的输出特征图。随后，剪枝后的主干网络可与原始网络头部结合进行预测。我们通过将梯度反向传播至噪声图像来生成合成图像，同时采用L1剪枝方法对主干网络进行剪枝。实验表明，由于仅对主干网络进行剪枝，本方法具有任务无关性。通过在二维人体姿态估计、目标检测和图像分类任务上的评估，我们发现该方法与未剪枝模型相比展现出有竞争力的性能。相关代码已开源至 https://github.com/holzbock/dfbf。