Regularizing Deep Neural Networks (DNNs) is essential for improving generalizability and preventing overfitting. Fixed penalty methods, though common, lack adaptability and suffer from hyperparameter sensitivity. In this paper, we propose a novel approach to DNN regularization by framing the training process as a constrained optimization problem. Where the data fidelity term is the minimization objective and the regularization terms serve as constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method to achieve a more flexible and efficient regularization mechanism. Our approach extends beyond black-box regularization, demonstrating significant improvements in white-box models, where weights are often subject to hard constraints to ensure interpretability. Experimental results on image-based classification on MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our approach. SAL consistently achieves higher Accuracy while also achieving better constraint satisfaction, thus showcasing its potential for optimizing DNNs under constrained settings.

翻译：深度神经网络的正则化对于提升泛化能力和防止过拟合至关重要。固定惩罚方法虽然常见，但缺乏适应性且对超参数敏感。本文通过将训练过程构建为约束优化问题，提出了一种新颖的DNN正则化方法。其中数据保真项为最小化目标，而正则化项则作为约束条件。随后，我们采用随机增广拉格朗日方法实现更灵活高效的正则化机制。本方法超越黑箱正则化范畴，在白箱模型中展现出显著改进——此类模型的权重常需硬约束以保证可解释性。基于MNIST、CIFAR10和CIFAR100数据集的图像分类实验结果验证了本方法的有效性。SAL在实现更高精确度的同时，还能更好地满足约束条件，从而展示了其在约束条件下优化深度神经网络的潜力。