Utilizing large-scale pretrained models is a well-known strategy to enhance performance on various target tasks. It is typically achieved through fine-tuning pretrained models on target tasks. However, na\"{\i}ve fine-tuning may not fully leverage knowledge embedded in pretrained models. In this study, we introduce a novel fine-tuning method, called stochastic cross-attention (StochCA), specific to Transformer architectures. This method modifies the Transformer's self-attention mechanism to selectively utilize knowledge from pretrained models during fine-tuning. Specifically, in each block, instead of self-attention, cross-attention is performed stochastically according to the predefined probability, where keys and values are extracted from the corresponding block of a pretrained model. By doing so, queries and channel-mixing multi-layer perceptron layers of a target model are fine-tuned to target tasks to learn how to effectively exploit rich representations of pretrained models. To verify the effectiveness of StochCA, extensive experiments are conducted on benchmarks in the areas of transfer learning and domain generalization, where the exploitation of pretrained models is critical. Our experimental results show the superiority of StochCA over state-of-the-art approaches in both areas. Furthermore, we demonstrate that StochCA is complementary to existing approaches, i.e., it can be combined with them to further improve performance. Our code is available at https://github.com/daintlab/stochastic_cross_attention

翻译：利用大规模预训练模型是提升各类目标任务性能的常见策略，通常通过在目标任务上微调预训练模型来实现。然而，朴素微调可能无法充分利用预训练模型中蕴含的知识。本研究提出了一种名为随机交叉注意力（StochCA）的新型微调方法，该方法专为Transformer架构设计。该方法通过修改Transformer的自注意力机制，在微调过程中选择性利用预训练模型的知识。具体而言，在每个模块中，依据预设概率随机执行交叉注意力替代自注意力，其中键和值从预训练模型的对应模块中提取。通过这种方式，目标模型的查询向量和通道混合多层感知机层被微调以适应目标任务，从而学习如何有效利用预训练模型的丰富表征。为验证StochCA的有效性，我们在迁移学习和领域泛化这两个依赖预训练模型的关键基准任务上开展了大量实验。实验结果表明，StochCA在这两个领域均优于当前最优方法。此外，我们证明了StochCA与现有方法具有互补性，即可以与之结合以进一步提升性能。代码可从https://github.com/daintlab/stochastic_cross_attention获取。