Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. In machine learning approaches based on linear models, it is well known that there exists a connecting path between the sparsest solution in terms of the $\ell^1$ norm (i.e., zero weights) and the non-regularized solution, which is called the regularization path. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity ($\ell^1$ norm) as two conflicting criteria and solving the resulting multiobjective optimization problem. However, due to the non-smoothness of the $\ell^1$ norm and the high number of parameters, this approach is not very efficient from a computational perspective. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization.

翻译：稀疏性是深度神经网络（DNNs）中一个高度期望的特性，因为它能确保数值效率，提高模型的可解释性（由于相关特征数量较少），并增强鲁棒性。在基于线性模型的机器学习方法中，众所周知，在基于$\ell^1$范数（即零权重）的最稀疏解与非正则化解之间存在一条连接路径，称为正则化路径。最近，有人首次尝试将正则化路径的概念扩展到深度神经网络，其方法是将经验损失和稀疏性（$\ell^1$范数）视为两个相互冲突的准则，并求解由此产生的多目标优化问题。然而，由于$\ell^1$范数的不光滑性以及参数数量庞大，该方法在计算上效率不高。为克服这一局限性，我们提出了一种算法，能够非常高效地近似计算上述目标的整个帕累托前沿。我们使用确定性和随机梯度展示了数值示例。此外，我们证明了对正则化路径的了解能够实现一种泛化性能良好的网络参数化。