Abstractive summarization is the process of generating a summary given a document as input. Although significant progress has been made, the factual inconsistency between the document and the generated summary still limits its practical applications. Previous work found that the probabilities assigned by the generation model reflect its preferences for the generated summary, including the preference for factual consistency, and the preference for the language or knowledge prior as well. To separate the preference for factual consistency, we propose an unsupervised framework named CoP by controlling the preference of the generation model with the help of prompt. More specifically, the framework performs an extra inference step in which a text prompt is introduced as an additional input. In this way, another preference is described by the generation probability of this extra inference process. The difference between the above two preferences, i.e. the difference between the probabilities, could be used as measurements for detecting factual inconsistencies. Interestingly, we found that with the properly designed prompt, our framework could evaluate specific preferences and serve as measurements for fine-grained categories of inconsistency, such as entity-related inconsistency, coreference-related inconsistency, etc. Moreover, our framework could also be extended to the supervised setting to learn better prompt from the labeled data as well. Experiments show that our framework achieves new SOTA results on three factual inconsistency detection tasks.

翻译：摘要：抽象式摘要是指以文档为输入生成摘要的过程。尽管已取得显著进展，但文档与生成摘要之间的事实不一致性仍然限制了其实际应用。先前研究发现，生成模型分配的概率反映了其对生成摘要的偏好，包括对事实一致性的偏好，以及对语言或知识先验的偏好。为了分离对事实一致性的偏好，我们提出了一种名为CoP的无监督框架，通过借助提示控制生成模型的偏好来实现这一目标。具体而言，该框架执行一个额外的推理步骤，在该步骤中引入文本提示作为附加输入。通过这种方式，该额外推理过程的生成概率描述了另一种偏好。上述两种偏好之间的差异，即概率之差，可作为检测事实不一致性的度量指标。有趣的是，我们发现通过精心设计的提示，该框架能够评估特定偏好，并作为细粒度不一致类别（如实体相关不一致、共指相关不一致等）的度量。此外，该框架还可扩展至有监督设置，以从标注数据中学习更好的提示。实验表明，我们的框架在三个事实不一致性检测任务上达到了新的最优结果。