In this paper, we consider the problem of learning a linear regression model on a data domain of interest (target) given few samples. To aid learning, we are provided with a set of pre-trained regression models that are trained on potentially different data domains (sources). Assuming a representation structure for the data generating linear models at the sources and the target domains, we propose a representation transfer based learning method for constructing the target model. The proposed scheme is comprised of two phases: (i) utilizing the different source representations to construct a representation that is adapted to the target data, and (ii) using the obtained model as an initialization to a fine-tuning procedure that re-trains the entire (over-parameterized) regression model on the target data. For each phase of the training method, we provide excess risk bounds for the learned model compared to the true data generating target model. The derived bounds show a gain in sample complexity for our proposed method compared to the baseline method of not leveraging source representations when achieving the same excess risk, therefore, theoretically demonstrating the effectiveness of transfer learning for linear regression.

翻译：本文研究在目标数据域仅有少量样本的情况下，学习线性回归模型的问题。为辅助学习，我们获得一组在不同数据域（源域）上训练的预训练回归模型。假设源域与目标域数据生成线性模型具有表示结构，我们提出一种基于表示迁移的学习方法以构建目标模型。所提方案包含两个阶段：（i）利用不同源域表示构建适应目标数据的表示；（ii）将所得模型作为微调过程的初始值，在目标数据上重新训练整个（过参数化）回归模型。针对训练方法的每个阶段，我们给出了学习模型相对于真实数据生成目标模型的超额风险界。推导出的界限表明：在实现相同超额风险时，与不利用源域表示的基准方法相比，我们提出的方法在样本复杂度上具有优势，从而从理论上证明了线性回归迁移学习的有效性。