In deep learning, often the training process finds an interpolator (a solution with 0 training loss), but the test loss is still low. This phenomenon, known as benign overfitting, is a major mystery that received a lot of recent attention. One common mechanism for benign overfitting is implicit regularization, where the training process leads to additional properties for the interpolator, often characterized by minimizing certain norms. However, even for a simple sparse linear regression problem $y = \beta^{*\top} x +\xi$ with sparse $\beta^*$, neither minimum $\ell_1$ or $\ell_2$ norm interpolator gives the optimal test loss. In this work, we give a different parametrization of the model which leads to a new implicit regularization effect that combines the benefit of $\ell_1$ and $\ell_2$ interpolators. We show that training our new model via gradient descent leads to an interpolator with near-optimal test loss. Our result is based on careful analysis of the training dynamics and provides another example of implicit regularization effect that goes beyond norm minimization.

翻译：在深度学习中，训练过程常常会找到一个插值器（训练损失为零的解），但测试损失仍然很低。这种被称为良性过拟合的现象是一个重大谜团，近期受到了大量关注。良性过拟合的一种常见机制是隐式正则化，即训练过程会赋予插值器额外的性质，通常表现为最小化某些范数。然而，即使对于简单的稀疏线性回归问题 $y = \beta^{*\top} x +\xi$（其中 $\beta^*$ 是稀疏的），最小 $\ell_1$ 范数或最小 $\ell_2$ 范数插值器都无法给出最优的测试损失。在本工作中，我们给出了模型的一种不同参数化形式，从而产生了一种新的隐式正则化效应，该效应结合了 $\ell_1$ 和 $\ell_2$ 插值器的优点。我们证明，通过梯度下降训练我们的新模型会得到一个具有近最优测试损失的插值器。我们的结果基于对训练动态的细致分析，并提供了超越范数最小化的隐式正则化效应的另一个实例。