Despite the importance of denoising in modern machine learning and ample empirical work on supervised denoising, its theoretical understanding is still relatively scarce. One concern about studying supervised denoising is that one might not always have noiseless training data from the test distribution. It is more reasonable to have access to noiseless training data from a different dataset than the test dataset. Motivated by this, we study supervised denoising and noisy-input regression under distribution shift. We add three considerations to increase the applicability of our theoretical insights to real-life data and modern machine learning. First, while most past theoretical work assumes that the data covariance matrix is full-rank and well-conditioned, empirical studies have shown that real-life data is approximately low-rank. Thus, we assume that our data matrices are low-rank. Second, we drop independence assumptions on our data. Third, the rise in computational power and dimensionality of data have made it important to study non-classical regimes of learning. Thus, we work in the non-classical proportional regime, where data dimension $d$ and number of samples $N$ grow as $d/N = c + o(1)$. For this setting, we derive general test error expressions for both denoising and noisy-input regression, and study when overfitting the noise is benign, tempered or catastrophic. We show that the test error exhibits double descent under general distribution shift, providing insights for data augmentation and the role of noise as an implicit regularizer. We also perform experiments using real-life data, where we match the theoretical predictions with under 1% MSE error for low-rank data.

翻译：尽管去噪在现代机器学习中至关重要，且关于监督去噪的实证研究日益丰富，但其理论基础仍相对匮乏。监督去噪研究面临的一个核心挑战在于：实际场景中往往难以获取与测试集同分布的无噪声训练数据。更为现实的情形是，我们能从与测试数据集不同的源数据集中获取无噪声训练样本。基于此，我们研究了分布偏移下的监督去噪与噪声输入回归问题。为增强理论洞见对真实数据及现代机器学习的适用性，我们引入三项考量：首先，尽管既往理论工作多假设数据协方差矩阵满秩且良态，但实证研究表明真实数据近似低秩，因此我们假定数据矩阵具有低秩特性；其次，我们摒弃了数据独立性假设；第三，随着计算能力与数据维度的提升，研究非经典学习范式愈发重要，故我们采用非经典比例极限框架，即数据维度$d$与样本量$N$满足$d/N = c + o(1)$的同步增长关系。在此设定下，我们推导出去噪与噪声输入回归的泛化误差解析表达式，系统分析了过拟合噪声呈现良性、温和或灾难性效应的条件。研究表明，在通用分布偏移场景下，测试误差呈现双重下降现象，为数据增强方法及噪声隐式正则化机制提供了新见解。基于真实数据的实验表明，在低秩数据场景下，理论预测与实测均方误差（MSE）的偏差小于1%。