Pruning schemes have been widely used in practice to reduce the complexity of trained models with a massive number of parameters. Several practical studies have shown that pruning an overparameterized model and fine-tuning generalizes well to new samples. Although the above pipeline, which we refer to as pruning + fine-tuning, has been extremely successful in lowering the complexity of trained models, there is very little known about the theory behind this success. In this paper we address this issue by investigating the pruning + fine-tuning framework on the overparameterized matrix sensing problem, with the ground truth denoted $U_\star \in \mathbb{R}^{d \times r}$ and the overparameterized model $U \in \mathbb{R}^{d \times k}$ with $k \gg r$. We study the approximate local minima of the empirical mean square error, augmented with a smooth version of a group Lasso regularizer, $\sum_{i=1}^k \| U e_i \|_2$ and show that pruning the low $\ell_2$-norm columns results in a solution $U_{\text{prune}}$ which has the minimum number of columns $r$, yet is close to the ground truth in training loss. Initializing the subsequent fine-tuning phase from $U_{\text{prune}}$, the resulting solution converges linearly to a generalization error of $O(\sqrt{rd/n})$ ignoring lower order terms, which is statistically optimal. While our analysis provides insights into the role of regularization in pruning, we also show that running gradient descent in the absence of regularization results in models which {are not suitable for greedy pruning}, i.e., many columns could have their $\ell_2$ norm comparable to that of the maximum. Lastly, we extend our results for the training and pruning of two-layer neural networks with quadratic activation functions. Our results provide the first rigorous insights on why greedy pruning + fine-tuning leads to smaller models which also generalize well.

翻译：剪枝策略已被广泛用于降低包含大量参数的已训练模型的复杂度。多项实践研究表明，对过参数化模型进行剪枝后微调，能够在新样本上表现出良好的泛化性能。尽管这种被称为"剪枝+微调"的流程在降低模型复杂度方面取得了显著成功，但其理论机理尚不明确。本文针对过参数化矩阵感知问题研究该框架，其中真实信号记为$U_\star \in \mathbb{R}^{d \times r}$，过参数化模型为$U \in \mathbb{R}^{d \times k}$（$k \gg r$）。我们分析了带平滑分组Lasso正则化项$\sum_{i=1}^k \| U e_i \|_2$的经验均方误差的近似局部极小值，证明剪除低$\ell_2$范数列后得到的解$U_{\text{prune}}$具有最小列数$r$，同时在训练损失上逼近真实信号。以$U_{\text{prune}}$初始化后续微调阶段，所得解线性收敛至$O(\sqrt{rd/n})$（忽略低阶项）的泛化误差，该误差在统计上达到最优。我们的分析不仅揭示了正则化在剪枝中的作用，还表明未正则化时梯度下降产生的模型"不适合贪心剪枝"，即多数列的$\ell_2$范数与最大范数相当。最后，我们将结果拓展至带二次激活函数的两层神经网络的训练与剪枝。本研究首次严格证明为何贪心剪枝+微调能产生具备良好泛化性能的紧凑模型。