Classical wisdom suggests that estimators should avoid fitting noise to achieve good generalization. In contrast, modern overparameterized models can yield small test error despite interpolating noise -- a phenomenon often called "benign overfitting" or "harmless interpolation". This paper argues that the degree to which interpolation is harmless hinges upon the strength of an estimator's inductive bias, i.e., how heavily the estimator favors solutions with a certain structure: while strong inductive biases prevent harmless interpolation, weak inductive biases can even require fitting noise to generalize well. Our main theoretical result establishes tight non-asymptotic bounds for high-dimensional kernel regression that reflect this phenomenon for convolutional kernels, where the filter size regulates the strength of the inductive bias. We further provide empirical evidence of the same behavior for deep neural networks with varying filter sizes and rotational invariance.

翻译：经典观点认为，估计器应避免拟合噪声以实现良好泛化。相比之下，现代过参数化模型尽管插值噪声仍能取得较小测试误差——这一现象常被称为“良性过拟合”或“无害插值”。本文论证，插值无害的程度取决于估计器归纳偏置的强度，即估计器偏向具有特定结构解的程度：强归纳偏置阻止无害插值，而弱归纳偏置甚至需要拟合噪声才能实现良好泛化。我们的主要理论结果为高维核回归建立了紧致的非渐近界，该界反映了卷积核中的这一现象，其中滤波器尺寸调节归纳偏置的强度。我们进一步为具有不同滤波器尺寸和旋转不变性的深度神经网络提供了相同行为的经验证据。