Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval .

翻译：近年来，大语言模型评估已成为热点研究领域。大语言模型评估的三个核心问题是"评估什么、在哪里评估以及如何评估"。现有研究主要聚焦前两个问题，即测试时应向大语言模型提供何种任务及其需要处理的知识类型。对于第三个问题——评估标准、评估者类型、评分方式及排名机制，目前尚缺乏充分探讨。本文通过对比人工评估与自动评估中的多种标准，结合现场评估、众包评估、公共标注者及GPT-4，采用差异化评分方法与排名体系，对评估方法展开系统分析。我们构建了新数据集LLMEval，在20个大语言模型上开展评估。共计2,186人参与，生成243,337条人工标注与57,511条自动评估结果。通过对不同设置进行比较分析，我们总结出10项结论，可为未来大语言模型评估提供参考。数据集与结果已在https://github.com/llmeval 公开。