Despite the importance of denoising in modern machine learning and ample empirical work on supervised denoising, its theoretical understanding is still relatively scarce. One concern about studying supervised denoising is that one might not always have noiseless training data from the test distribution. It is more reasonable to have access to noiseless training data from a different dataset than the test dataset. Motivated by this, we study supervised denoising and noisy-input regression under distribution shift. We add three considerations to increase the applicability of our theoretical insights to real-life data and modern machine learning. First, while most past theoretical work assumes that the data covariance matrix is full-rank and well-conditioned, empirical studies have shown that real-life data is approximately low-rank. Thus, we assume that our data matrices are low-rank. Second, we drop independence assumptions on our data. Third, the rise in computational power and dimensionality of data have made it important to study non-classical regimes of learning. Thus, we work in the non-classical proportional regime, where data dimension $d$ and number of samples $N$ grow as $d/N = c + o(1)$. For this setting, we derive data-dependent, instance specific expressions for the test error for both denoising and noisy-input regression, and study when overfitting the noise is benign, tempered or catastrophic. We show that the test error exhibits double descent under general distribution shift, providing insights for data augmentation and the role of noise as an implicit regularizer. We also perform experiments using real-life data, where we match the theoretical predictions with under 1\% MSE error for low-rank data.

翻译：尽管去噪在现代机器学习中具有重要地位，且关于监督去噪的实证研究丰富，但其理论理解仍相对匮乏。研究监督去噪的一个担忧在于，我们未必总是能从测试分布中获得无噪声训练数据。更合理的情形是，使用来自不同于测试数据集的另一数据集的无噪声训练数据。受此启发，我们研究了分布偏移下的监督去噪与含噪输入回归。为提升理论见解对真实数据及现代机器学习的适用性，我们引入三项考量：第一，尽管过往理论工作多假设数据协方差矩阵满秩且条件良好，但实证研究表明真实数据近似低秩，因此我们假设数据矩阵为低秩；第二，我们摒弃对数据的独立性假设；第三，计算能力与数据维度的提升使得研究非经典学习范式变得重要，因此我们在非经典比例极限框架下开展工作，即数据维度 $d$ 与样本数 $N$ 满足 $d/N = c + o(1)$。针对此设定，我们推导出去噪与含噪输入回归的测试误差的数据依赖型实例特定表达式，并研究噪声过拟合何时呈现良性的、温和的或灾难性的行为。我们发现，在广义分布偏移下测试误差呈现双重下降现象，这为数据增强及噪声作为隐式正则化项的作用提供了洞见。我们还使用真实数据进行了实验，对于低秩数据，理论预测与均方误差低于1%的实验结果完美匹配。