Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. However, selecting the optimal pre-trained model from a diverse pool for a specific downstream task remains a challenge. Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels, but they overlook the impact of underlying representation dynamics during fine-tuning, leading to unreliable results, especially for self-supervised models. In this paper, we present an insightful physics-inspired approach named PED to address these challenges. We reframe the challenge of model selection through the lens of potential energy and directly model the interaction forces that influence fine-tuning dynamics. By capturing the motion of dynamic representations to decline the potential energy within a force-driven physical model, we can acquire an enhanced and more stable observation for estimating transferability. The experimental results on 10 downstream tasks and 12 self-supervised models demonstrate that our approach can seamlessly integrate into existing ranking techniques and enhance their performances, revealing its effectiveness for the model selection task and its potential for understanding the mechanism in transfer learning. Code will be available at https://github.com/lixiaotong97/PED.

翻译：迁移学习因预训练深度学习模型的广泛可用性而成为计算机视觉任务中的关键方法。然而，从多样化的预训练模型池中为特定下游任务选择最优模型仍具挑战性。现有衡量预训练模型迁移性的方法依赖于编码静态特征与任务标签之间的统计相关性，但忽略了微调过程中潜在表征动态的影响，导致结果不可靠，尤其对自监督模型而言。本文提出一种受物理学启发的深刻方法PED以应对这些挑战。我们通过势能视角重新定义模型选择问题，并直接建模影响微调动态的相互作用力。通过捕捉动态表征在力驱动物理模型中降低势能的运动过程，我们可获得增强且更稳定的观测结果以估计迁移性。在10个下游任务和12个自监督模型上的实验表明，我们的方法可无缝集成到现有排序技术中并提升其性能，揭示了其在模型选择任务中的有效性及其理解迁移学习机制的潜力。代码将发布于 https://github.com/lixiaotong97/PED。