Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 60.5% higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at https://github.com/renll/C-PMI.

翻译：现有的无参考轮次级对话评估指标未能充分捕捉用户与系统间的交互，导致其与人工评估的相关性普遍较低。针对这一问题，我们提出一种新型模型无关方法，利用条件逐点互信息（C-PMI）基于给定评估维度衡量系统与用户之间的轮次级交互。在广泛使用的FED对话评估数据集上的实验结果表明，与现有评估系统相比，我们的方法显著提升了与人工判断的相关性。通过将基于负对数似然的评分器替换为我们提出的C-PMI评分器，FED评估指标的平均斯皮尔曼相关系数相对提升了60.5%。我们的代码已在https://github.com/renll/C-PMI公开。