To fully leverage the advantages of large-scale pre-trained language models (PLMs) on downstream tasks, it has become a ubiquitous adaptation paradigm to fine-tune the entire parameters of PLMs. However, this paradigm poses issues of inefficient updating and resource over-consuming for fine-tuning in data-scarce and resource-limited scenarios, because of the large scale of parameters in PLMs. To alleviate these concerns, in this paper, we propose a parameter-efficient fine-tuning method HiFi, that is, only the highly informative and strongly correlated attention heads for the specific task are fine-tuned. To search for those significant attention heads, we develop a novel framework to analyze the effectiveness of heads. Specifically, we first model the relationship between heads into a graph from two perspectives of information richness and correlation, and then apply PageRank algorithm to determine the relative importance of each head. Extensive experiments on the GLUE benchmark demonstrate the effectiveness of our method, and show that HiFi obtains state-of-the-art performance over the prior baselines.

翻译：为在下游任务中充分利用大规模预训练语言模型（PLMs）的优势，微调PLMs的全部参数已成为一种普遍采用的适配范式。然而，由于PLMs参数量巨大，该范式在数据稀缺和资源受限的场景下会引发更新效率低下及资源过度消耗的问题。为缓解这些问题，本文提出一种参数高效微调方法HiFi，即仅针对特定任务微调那些信息丰富且相关性强的注意力头。为搜索这些关键注意力头，我们开发了一种新颖的框架来分析注意力头的有效性。具体而言，首先从信息丰富度和相关性两个角度将注意力头之间的关系建模为图，然后应用PageRank算法确定每个注意力头的相对重要性。在GLUE基准上的大量实验证明了该方法的有效性，并表明HiFi在先前基线方法上取得了最优性能。