Based on the remarkable achievements of pre-trained language models in abstractive summarization, the copying mechanism has proved helpful by improving the factuality, stability, and overall performance. This work proposes PROM, a new PhRase-level cOpying Mechanism that enhances attention on n-grams, which can be applied to zero-shot summarization with pre-training. PROM adds an indicator layer to explicitly pick up tokens in n-gram that can be copied from the source, and calculates an auxiliary loss for the copying prediction. Empirical studies show that PROM makes significant improvements in fine-tuning on benchmarks. In zero-shot setting, PROM is utilized in the self-supervised pre-training on raw corpora and provides new general baselines on a wide range of summarization datasets. Further analysis shows that PROM performs more reasonable copying and contributes to faithfulness.

翻译：基于预训练语言模型在抽象式摘要任务中取得的显著成就，复制机制通过提升事实性、稳定性和整体性能被证实具有实用价值。本文提出PROM——一种新型短语级复制机制，通过增强对n-gram的注意力机制，可应用于零样本摘要的预训练场景。该方法通过添加指示层显式识别源文本中可被复制的n-gram token，并计算辅助损失函数以优化复制预测。实验表明，PROM在微调基准测试中实现了显著性能提升。在零样本设置下，PROM被应用于原始语料的自监督预训练，为多个摘要数据集提供了新的通用基线。进一步分析显示，PROM能实现更合理的复制行为，并有助于提升摘要的忠实度。