In deep learning, often the training process finds an interpolator (a solution with 0 training loss), but the test loss is still low. This phenomenon, known as benign overfitting, is a major mystery that received a lot of recent attention. One common mechanism for benign overfitting is implicit regularization, where the training process leads to additional properties for the interpolator, often characterized by minimizing certain norms. However, even for a simple sparse linear regression problem $y = \beta^{*\top} x +\xi$ with sparse $\beta^*$, neither minimum $\ell_1$ or $\ell_2$ norm interpolator gives the optimal test loss. In this work, we give a different parametrization of the model which leads to a new implicit regularization effect that combines the benefit of $\ell_1$ and $\ell_2$ interpolators. We show that training our new model via gradient descent leads to an interpolator with near-optimal test loss. Our result is based on careful analysis of the training dynamics and provides another example of implicit regularization effect that goes beyond norm minimization.

翻译：在深度学习中，训练过程通常能找到插值器（训练损失为零的解），但测试损失仍然很低。这一被称为“良性过拟合”的现象是近期备受关注的主要谜团之一。良性过拟合的一种常见机制是隐式正则化，即训练过程为插值器赋予额外特性，通常表现为最小化特定范数。然而，即便对于稀疏线性回归问题 $y = \beta^{*\top} x +\xi$（其中 $\beta^*$ 是稀疏的），最小 $\ell_1$ 范数插值器和最小 $\ell_2$ 范数插值器都无法给出最优测试损失。本文提出了一种不同的模型参数化方法，由此产生了一种新的隐式正则化效应，该效应融合了 $\ell_1$ 和 $\ell_2$ 插值器的优势。我们证明，通过梯度下降训练新模型能够得到一个具有近最优测试损失的插值器。这一结果基于对训练动态的细致分析，并提供了超越范数最小化的隐式正则化效应的又一实例。