Transfer learning aims to improve the performance of target tasks by transferring knowledge acquired in source tasks. The standard approach is pre-training followed by fine-tuning or linear probing. Especially, selecting a proper source domain for a specific target domain under predefined tasks is crucial for improving efficiency and effectiveness. It is conventional to solve this problem via estimating transferability. However, existing methods can not reach a trade-off between performance and cost. To comprehensively evaluate estimation methods, we summarize three properties: stability, reliability and efficiency. Building upon them, we propose Principal Gradient Expectation(PGE), a simple yet effective method for assessing transferability. Specifically, we calculate the gradient over each weight unit multiple times with a restart scheme, and then we compute the expectation of all gradients. Finally, the transferability between the source and target is estimated by computing the gap of normalized principal gradients. Extensive experiments show that the proposed metric is superior to state-of-the-art methods on all properties.

翻译：迁移学习旨在通过迁移源任务中获取的知识来提升目标任务性能。其标准方法为预训练后接微调或线性探测。特别地，在预定义任务下为特定目标域选择适当的源域对于提升效率与效果至关重要。通常通过估计可迁移性来解决这一问题。然而，现有方法无法在性能与成本之间取得平衡。为综合评估各类估计方法，我们总结了三种特性：稳定性、可靠性与高效性。基于此，我们提出主梯度期望（Principal Gradient Expectation, PGE）——一种简单而有效的可迁移性评估方法。具体而言，我们通过重启策略多次计算每个权重单元上的梯度，随后计算所有梯度的期望值。最后，通过计算归一化主梯度的差距来估计源域与目标域之间的可迁移性。大量实验表明，所提指标在所有特性上均优于现有最优方法。