Keyphrase generation aims at generating topical phrases from a given text either by copying from the original text (present keyphrases) or by producing new keyphrases (absent keyphrases) that capture the semantic meaning of the text. Encoder-decoder models are most widely used for this task because of their capabilities for absent keyphrase generation. However, there has been little to no analysis on the performance and behavior of such models for keyphrase generation. In this paper, we study various tendencies exhibited by three strong models: T5 (based on a pre-trained transformer), CatSeq-Transformer (a non-pretrained Transformer), and ExHiRD (based on a recurrent neural network). We analyze prediction confidence scores, model calibration, and the effect of token position on keyphrases generation. Moreover, we motivate and propose a novel metric framework, SoftKeyScore, to evaluate the similarity between two sets of keyphrases by using softscores to account for partial matching and semantic similarity. We find that SoftKeyScore is more suitable than the standard F1 metric for evaluating two sets of given keyphrases.

翻译：关键词生成旨在从给定文本中生成主题性短语，既可以通过从原文复制（现有关键词），也可以生成能够捕捉文本语义的新关键词（缺失关键词）。编码器-解码器模型因其生成缺失关键词的能力而被最广泛地应用于此任务。然而，目前对于这类模型在关键词生成中的性能和行为的分析几乎空白。本文研究了三种强模型表现出的各种趋势：T5（基于预训练Transformer）、CatSeq-Transformer（非预训练Transformer）和ExHiRD（基于循环神经网络）。我们分析了预测置信度分数、模型校准以及标记位置对关键词生成的影响。此外，我们提出并论证了一种新颖的评估指标框架SoftKeyScore，通过使用软分数来考虑部分匹配和语义相似性，从而评估两组关键词之间的相似度。我们发现，与标准F1指标相比，SoftKeyScore更适合评估给定的两组关键词。