The computing cost and memory demand of deep learning pipelines have grown fast in recent years and thus a variety of pruning techniques have been developed to reduce model parameters. The majority of these techniques focus on reducing inference costs by pruning the network after a pass of full training. A smaller number of methods address the reduction of training costs, mostly based on compressing the network via low-rank layer factorizations. Despite their efficiency for linear layers, these methods fail to effectively handle convolutional filters. In this work, we propose a low-parametric training method that factorizes the convolutions into tensor Tucker format and adaptively prunes the Tucker ranks of the convolutional kernel during training. Leveraging fundamental results from geometric integration theory of differential equations on tensor manifolds, we obtain a robust training algorithm that provably approximates the full baseline performance and guarantees loss descent. A variety of experiments against the full model and alternative low-rank baselines are implemented, showing that the proposed method drastically reduces the training costs, while achieving high performance, comparable to or better than the full baseline, and consistently outperforms competing low-rank approaches.

翻译：深度学习管道的计算成本和内存需求近年来迅速增长，因此已开发出多种剪枝技术来减少模型参数。大多数技术侧重于通过在全训练阶段之后对网络进行剪枝来降低推理成本。少数方法解决了训练成本降低的问题，主要基于通过低秩层分解来压缩网络。尽管这些方法对线性层有效，但无法高效处理卷积滤波器。本文提出一种低参数训练方法，将卷积分解为张量Tucker格式，并在训练过程中自适应地剪枝卷积核的Tucker秩。利用张量流形上微分方程几何积分理论的基本结果，我们获得了一种稳健的训练算法，该算法可证明地逼近完整基线性能并保证损失下降。针对完整模型和替代低秩基线进行了多种实验，结果表明所提方法显著降低了训练成本，同时实现了与完整基线相当甚至更优的高性能，并且始终优于竞争性低秩方法。