Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the `where' and `how' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.

翻译：大型语言模型（LLMs）因其在各类应用中表现出的前所未有的性能，正日益受到学术界和工业界的广泛关注。随着LLMs在研究和日常使用中扮演着愈发重要的角色，其评估工作也显得尤为关键——不仅需要在任务层面进行，更需在社会层面展开，以更好地理解其潜在风险。过去数年间，研究者们已从多角度对LLMs进行了大量考察。本文对LLMs的评估方法进行了全面综述，聚焦三个核心维度：评估什么、在哪里评估以及如何评估。首先，我们从评估任务的视角进行概述，涵盖通用自然语言处理任务、推理、医学应用、伦理、教育、自然科学与社会科学、智能体应用及其他领域。其次，我们通过深入分析评估方法与基准来回答"在哪里"和"如何"的问题——这些要素是衡量LLMs性能的关键组成部分。接着，我们总结了LLMs在不同任务中的成功与失败案例。最后，我们探讨了LLMs评估面临的若干未来挑战。旨在为LLMs评估领域的研究者提供宝贵见解，从而助力开发更高效的LLMs。我们的核心观点是：评估应被视为一门基础学科，用以更好地支撑LLMs的发展。本文相关开源资料持续维护于：https://github.com/MLGroupJLU/LLM-eval-survey。