The impressive performance of deep learning architectures is associated with a massive increase in model complexity. Millions of parameters need to be tuned, with training and inference time scaling accordingly, together with energy consumption. But is massive fine-tuning always necessary? In this paper, focusing on image classification, we consider a simple transfer learning approach exploiting pre-trained convolutional features as input for a fast-to-train kernel method. We refer to this approach as \textit{top-tuning} since only the kernel classifier is trained on the target dataset. In our study, we perform more than 3000 training processes focusing on 32 small to medium-sized target datasets, a typical situation where transfer learning is necessary. We show that the top-tuning approach provides comparable accuracy with respect to fine-tuning, with a training time between one and two orders of magnitude smaller. These results suggest that top-tuning is an effective alternative to fine-tuning in small/medium datasets, being especially useful when training time efficiency and computational resources saving are crucial.

翻译：深度学习架构的卓越性能伴随着模型复杂度的急剧增加。数百万个参数需要调优，训练与推理时间随之增加，能耗亦相应攀升。但大规模微调是否始终必要？本文聚焦图像分类任务，提出一种简单的迁移学习方法：利用预训练卷积特征作为快速训练核方法的输入。我们将该方法称为“Top-Tuning”——仅对目标数据集训练核分类器。研究围绕32个中小型目标数据集（迁移学习典型的必要场景）开展了超过3000次训练流程。结果表明，Top-Tuning在保证与微调相当精度的同时，训练时间减少了一个至两个数量级。这些发现证明：在中小型数据集场景下，Top-Tuning是微调的有效替代方案，尤其适用于训练时间效率与计算资源节约至关重要的场景。