Designing deep neural network classifiers that perform robustly on distributions differing from the available training data is an active area of machine learning research. However, out-of-distribution generalization for regression-the analogous problem for modeling continuous targets-remains relatively unexplored. To tackle this problem, we return to first principles and analyze how the closed-form solution for Ordinary Least Squares (OLS) regression is sensitive to covariate shift. We characterize the out-of-distribution risk of the OLS model in terms of the eigenspectrum decomposition of the source and target data. We then use this insight to propose a method for adapting the weights of the last layer of a pre-trained neural regression model to perform better on input data originating from a different distribution. We demonstrate how this lightweight spectral adaptation procedure can improve out-of-distribution performance for synthetic and real-world datasets.

翻译：设计能够对与可用训练数据分布不同的数据实现稳健性能的深度神经网络分类器，是机器学习研究的一个活跃领域。然而，针对连续目标建模的同类问题——分布外泛化回归任务，目前仍相对缺乏探索。为解决该问题，我们回归基本原理，分析普通最小二乘回归的闭式解如何对协变量偏移敏感。我们通过源数据与目标数据的特征谱分解来表征OLS模型的分布外风险，并基于此洞察提出一种方法，用于调整预训练神经回归模型最后层的权重，使其在源于不同分布的输入数据上表现更优。实验证明，这种轻量级光谱自适应过程能提升合成数据集与真实数据集上的分布外性能。