Coping with distributional shifts is an important part of transfer learning methods in order to perform well in real-life tasks. However, most of the existing approaches in this area either focus on an ideal scenario in which the data does not contain noises or employ a complicated training paradigm or model design to deal with distributional shifts. In this paper, we revisit the robustness of the minimum error entropy (MEE) criterion, a widely used objective in statistical signal processing to deal with non-Gaussian noises, and investigate its feasibility and usefulness in real-life transfer learning regression tasks, where distributional shifts are common. Specifically, we put forward a new theoretical result showing the robustness of MEE against covariate shift. We also show that by simply replacing the mean squared error (MSE) loss with the MEE on basic transfer learning algorithms such as fine-tuning and linear probing, we can achieve competitive performance with respect to state-of-the-art transfer learning algorithms. We justify our arguments on both synthetic data and 5 real-world time-series data.

翻译：处理分布偏移是迁移学习方法在现实任务中取得良好性能的重要组成部分。然而，该领域现有的大多数方法要么聚焦于数据不含噪声的理想场景，要么采用复杂的训练范式或模型设计来应对分布偏移。本文重新审视了最小误差熵（MEE）准则的鲁棒性——该准则是统计信号处理中广泛用于处理非高斯噪声的目标函数，并探究其在分布偏移普遍存在的现实迁移学习回归任务中的可行性与实用性。具体而言，我们提出了新的理论结果，证明了MEE对协变量偏移的鲁棒性。我们还表明，通过简单地将均方误差（MSE）损失替换为MEE，应用于微调和线性探测等基础迁移学习算法时，能够达到与当前最优迁移学习算法相媲美的性能。我们通过合成数据与5个真实世界时间序列数据验证了上述论点。