Measuring empathy in conversation can be challenging, as empathy is a complex and multifaceted psychological construct that involves both cognitive and emotional components. Human evaluations can be subjective, leading to inconsistent results. Therefore, there is a need for an automatic method for measuring empathy that reduces the need for human evaluations. In this paper, we proposed a novel approach EMP-EVAL, a simple yet effective automatic empathy evaluation method. The proposed technique takes the influence of Emotion, Cognitive and Emotional empathy. To the best knowledge, our work is the first to systematically measure empathy without the human-annotated provided scores. Experimental results demonstrate that our metrics can correlate with human preference, achieving comparable results with human judgments.

翻译：衡量对话中的同理心具有一定挑战性，因为同理心是一种涉及认知和情感成分的复杂多维心理建构。人工评估可能存在主观性，导致结果不一致。因此，需要一种减少对人工评估依赖的自动衡量同理心方法。本文提出了一种新颖方法EMP-EVAL，一种简单而有效的自动同理心评估方法。该技术综合考虑了情感同理心、认知同理心和情感同理心的影响。据我们所知，我们的工作是首次在没有人工标注评分的情况下系统衡量同理心。实验结果表明，我们的指标能够与人类偏好相关，达到与人工判断相当的结果。