Recent research has revealed that high compression of Deep Neural Networks (DNNs), e.g., massive pruning of the weight matrix of a DNN, leads to a severe drop in accuracy and susceptibility to adversarial attacks. Integration of network pruning into an adversarial training framework has been proposed to promote adversarial robustness. It has been observed that a highly pruned weight matrix tends to be ill-conditioned, i.e., increasing the condition number of the weight matrix. This phenomenon aggravates the vulnerability of a DNN to input noise. Although a highly pruned weight matrix is considered to be able to lower the upper bound of the local Lipschitz constant to tolerate large distortion, the ill-conditionedness of such a weight matrix results in a non-robust DNN model. To overcome this challenge, this work develops novel joint constraints to adjust the weight distribution of networks, namely, the Transformed Sparse Constraint joint with Condition Number Constraint (TSCNC), which copes with smoothing distribution and differentiable constraint functions to reduce condition number and thus avoid the ill-conditionedness of weight matrices. Furthermore, our theoretical analyses unveil the relevance between the condition number and the local Lipschitz constant of the weight matrix, namely, the sharply increasing condition number becomes the dominant factor that restricts the robustness of over-sparsified models. Extensive experiments are conducted on several public datasets, and the results show that the proposed constraints significantly improve the robustness of a DNN with high pruning rates.

翻译：近期研究表明，深度神经网络（DNN）的高度压缩（例如对DNN权重矩阵进行大规模剪枝）会导致精度严重下降并增加对对抗性攻击的敏感性。已有研究提出将网络剪枝集成到对抗性训练框架中以增强对抗鲁棒性。研究发现，高度剪枝后的权重矩阵往往呈现病态特性，即其条件数会增大。这一现象加剧了DNN对输入噪声的脆弱性。尽管高度剪枝的权重矩阵被认为能够降低局部Lipschitz常数的上界以容忍较大失真，但此类权重矩阵的病态性会导致DNN模型缺乏鲁棒性。为克服这一挑战，本研究提出了调整网络权重分布的新型联合约束方法——结合条件数约束的变换稀疏约束（TSCNC），该方法通过平滑分布和可微约束函数来处理条件数降低问题，从而避免权重矩阵的病态性。进一步的理论分析揭示了权重矩阵条件数与局部Lipschitz常数之间的内在关联：急剧增大的条件数成为限制过度稀疏模型鲁棒性的主导因素。在多个公开数据集上进行的广泛实验表明，所提出的约束方法能显著提升高剪枝率DNN的鲁棒性。