Evaluating Large Language Models' (LLMs) anthropomorphic capabilities has become increasingly important in contemporary discourse. Utilizing the emotion appraisal theory from psychology, we propose to evaluate the empathy ability of LLMs, i.e., how their feelings change when presented with specific situations. After a careful and comprehensive survey, we collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. Categorizing the situations into 36 factors, we conduct a human evaluation involving more than 1,200 subjects worldwide. With the human evaluation results as references, our evaluation includes five LLMs, covering both commercial and open-source models, including variations in model sizes, featuring the latest iterations, such as GPT-4 and LLaMA-2. We find that, despite several misalignments, LLMs can generally respond appropriately to certain situations. Nevertheless, they fall short in alignment with the emotional behaviors of human beings and cannot establish connections between similar situations. Our collected dataset of situations, the human evaluation results, and the code of our testing framework, dubbed EmotionBench, is made openly accessible via https://github.com/CUHK-ARISE/EmotionBench. We aspire to contribute to the advancement of LLMs regarding better alignment with the emotional behaviors of human beings, thereby enhancing their utility and applicability as intelligent assistants.

翻译：评估大语言模型（LLMs）的拟人化能力已成为当代研究的重要议题。基于心理学中的情绪评价理论，我们提出评估LLMs的共情能力，即当面对特定情境时其情感如何变化。经过全面系统的文献调研，我们构建了一个包含400余种情境的数据集，这些情境经实证可有效诱发本研究聚焦的八类核心情绪。将情境归入36个影响因子后，我们开展了涵盖全球1200余名被试的人工评估。以人工评估结果为参照，我们对五种LLMs进行了评测，涵盖商业与开源模型，包括不同参数规模的版本及最新迭代（如GPT-4和LLaMA-2）。研究发现，尽管存在若干偏差，LLMs总体上能对特定情境作出适当回应，但在情绪行为的人类对齐方面仍存不足，且无法建立相似情境间的关联。我们收集的情境数据集、人工评估结果及测试框架EmotionBench的代码均通过https://github.com/CUHK-ARISE/EmotionBench开源。期望本研究能推动LLMs在情绪行为人类对齐方面取得进展，从而提升其作为智能助手的实用性与适用性。