Given a set of pre-trained models, how can we quickly and accurately find the most useful pre-trained model for a downstream task? Transferability measurement is to quantify how transferable is a pre-trained model learned on a source task to a target task. It is used for quickly ranking pre-trained models for a given task and thus becomes a crucial step for transfer learning. Existing methods measure transferability as the discrimination ability of a source model for a target data before transfer learning, which cannot accurately estimate the fine-tuning performance. Some of them restrict the application of transferability measurement in selecting the best supervised pre-trained models that have classifiers. It is important to have a general method for measuring transferability that can be applied in a variety of situations, such as selecting the best self-supervised pre-trained models that do not have classifiers, and selecting the best transferring layer for a target task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure transferability. We view transferability as the generalization of a pre-trained model on a target task by measuring intra-class feature variance. Intra-class variance evaluates the adaptability of the model to a new task, which measures how transferable the model is. Compared to previous studies that estimate how discriminative the models are, intra-class variance is more accurate than those as it does not require an optimal feature extractor and classifier. Extensive experiments on real-world datasets show that TMI outperforms competitors for selecting the top-5 best models, and exhibits consistently better correlation in 13 out of 17 cases.

翻译：给定一组预训练模型，如何快速准确地找到对下游任务最有用的预训练模型？迁移性度量旨在量化在源任务上学习的预训练模型迁移到目标任务的可迁移程度，常用于快速排序给定任务的预训练模型，因此成为迁移学习的关键步骤。现有方法在迁移学习前将迁移性度量表示为源模型对目标数据的判别能力，这无法准确估计微调性能。部分方法限制了迁移性度量在具有分类器的监督预训练模型选择中的应用。因此，需要一种通用的迁移性度量方法，可适用于多种场景，例如选择无分类器的自监督预训练模型，以及为目标任务选择最佳迁移层。本文提出TMI（基于类内特征方差的迁移性度量），一种快速准确的迁移性度量算法。我们将迁移性视为通过评估类内特征方差衡量预训练模型在目标任务上的泛化能力。类内方差评估模型对新任务的适应性，从而度量模型的可迁移程度。与以往评估模型判别能力的研究相比，类内方差无需最优特征提取器和分类器，因此准确度更高。在真实数据集上的大量实验表明，TMI在Top-5最佳模型选择任务中优于对比方法，并在17个案例中的13个展现出更优的相关性。