The Transformer-based models with the multi-head self-attention mechanism are widely used in natural language processing, and provide state-of-the-art results. While the pre-trained language backbones are shown to implicitly capture certain linguistic knowledge, explicitly incorporating structure-aware features can bring about further improvement on the downstream tasks. However, such enhancement often requires additional neural components and increases training parameter size. In this work, we investigate the attention head selection and manipulation strategy for feature injection from a network pruning perspective, and conduct a case study on dialogue summarization. We first rank attention heads in a Transformer-based summarizer with layer-wise importance. We then select the underused heads through extensive analysis, and inject structure-aware features by manipulating the selected heads. Experimental results show that the importance-based head selection is effective for feature injection, and dialogue summarization can be improved by incorporating coreference information via head manipulation.

翻译：摘要：基于Transformer的多头自注意力机制模型广泛应用于自然语言处理，并取得了最先进的结果。虽然预训练语言骨干网络已被证明能隐式捕获某些语言知识，但显式融入结构感知特征可进一步提升下游任务性能。然而，这种增强通常需要额外的神经组件，并增加训练参数量。本研究从网络剪枝视角探讨用于特征注入的注意力头选择与操控策略，并以对话摘要为案例展开分析。我们首先通过层级重要性对基于Transformer的摘要模型中的注意力头进行排序，继而通过广泛分析筛选未充分利用的注意力头，并通过操控所选头注入结构感知特征。实验结果表明，基于重要性的头部选择对特征注入有效，且通过头部操控融入共指信息可提升对话摘要性能。