A major concern in using deep learning based generative models for document-grounded dialogs is the potential generation of responses that are not \textit{faithful} to the underlying document. Existing automated metrics used for evaluating the faithfulness of response with respect to the grounding document measure the degree of similarity between the generated response and the document's content. However, these automated metrics are far from being well aligned with human judgments. Therefore, to improve the measurement of faithfulness, we propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. PMI quantifies the extent to which the document influences the generated response -- with a higher PMI indicating a more faithful response. We build upon this idea to create a new decoding technique that incorporates PMI into the response generation process to predict more faithful responses. Our experiments on the BEGIN benchmark demonstrate an improved correlation of our metric with human evaluation. We also show that our decoding technique is effective in generating more faithful responses when compared to standard decoding techniques on a set of publicly available document-grounded dialog datasets.

翻译：在基于深度学习的生成模型用于文档对话时，主要问题是可能生成与底层文档不忠实（faithful）的回复。现有用于评估回复对基础文档忠实度的自动化指标，通过衡量生成回复与文档内容之间的相似度来评估。然而，这些自动化指标远未与人工判断良好对齐。因此，为了改进忠实度的度量，我们提出了一种新指标，该指标利用了生成回复与源文档之间的（条件）点互信息（Point-wise Mutual Information, PMI），并以对话为条件。PMI量化了文档对生成回复的影响程度——较高的PMI表示更忠实的回复。我们基于这一思想创建了一种新的解码技术，将PMI融入回复生成过程，以预测更忠实的回复。我们在BEGIN基准上的实验表明，我们的指标与人工评估的相关性有所提高。我们还证明，在一组公开可用的文档对话数据集上，与标准解码技术相比，我们的解码技术在生成更忠实的回复方面是有效的。