Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability of models (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a \emph{regularization path} connecting the sparsest solution in terms of the $\ell^1$ norm, i.e., zero weights and the non-regularized solution. Very recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity ($\ell^1$ norm) as two conflicting criteria and solving the resulting multiobjective optimization problem for low-dimensional DNN. However, due to the non-smoothness of the $\ell^1$ norm and the high number of parameters, this approach is not very efficient from a computational perspective for high-dimensional DNNs. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. To the best of our knowledge, this is the first algorithm to compute the regularization path for non-convex multiobjective optimization problems (MOPs) with millions of degrees of freedom.

翻译：稀疏性是深度神经网络（DNN）中一个非常理想的特征，因为它能保证数值效率，提升模型的可解释性（由于相关特征数量更少）以及鲁棒性。对于线性模型而言，众所周知存在一条 *正则化路径*，连接了在 $\ell^1$ 范数意义下的最稀疏解（即零权重）与未正则化解。最近，已有初步尝试通过将经验损失与稀疏性（$\ell^1$ 范数）视为两个相互冲突的目标，并求解由此产生的低维DNN多目标优化问题，将正则化路径的概念扩展到DNN领域。然而，由于 $\ell^1$ 范数的非光滑性以及参数数量庞大，这种方法对于高维DNN而言在计算上效率较低。为克服这一局限，我们提出了一种算法，能高效逼近上述目标下高维DNN（包含数百万个参数）的整个帕累托前沿。我们给出了使用确定性梯度与随机梯度的数值示例，并进一步证明了：对正则化路径的了解能够有助于得到泛化能力良好的网络参数化。据我们所知，这是首个针对拥有数百万自由度的非凸多目标优化问题（MOP）计算其正则化路径的算法。