Large language models (LLMs) garner significant attention for their unprecedented performance, leading to an increasing number of researches evaluating LLMs. However, these evaluation benchmarks are limited to assessing the instruction-following capabilities, overlooking the fundamental abilities that emerge during the pre-training stage. Previous subjective evaluation methods mainly reply on scoring by API models. However, in the absence of references, large models have shown limited ability to discern subtle differences. To bridge the gap, we propose F-Eval, a bilingual evaluation benchmark to evaluate the fundamental abilities, including expression, commonsense and logic. The tasks in F-Eval include multi-choice objective tasks, open-ended objective tasks, reference-based subjective tasks and reference-free subjective tasks. For reference-free subjective tasks, we devise new evaluation methods, serving as alternatives to scoring by API models. We conduct evaluations on 13 advanced LLMs. Results show that our evaluation methods show higher correlation coefficients and larger distinction than other evaluators. Additionally, we discuss the influence of different model sizes, dimensions, and normalization methods. We anticipate that F-Eval will facilitate the study of LLMs' fundamental abilities.

翻译：大型语言模型（LLM）因其前所未有的性能表现而受到广泛关注，相关评估研究也日益增多。然而，现有评估基准多局限于测试模型遵循指令的能力，忽视了在预训练阶段涌现的基础能力。先前的主观评估方法主要依赖API模型进行评分，但在缺乏参考答案的情况下，大模型对细微差异的辨别能力有限。为弥补这一不足，我们提出了F-Eval——一个用于评估基础能力的双语评估基准，涵盖表达能力、常识推理与逻辑推理三大维度。F-Eval包含多项选择型客观任务、开放式客观任务、基于参考答案的主观任务以及无参考答案的主观任务四类测评任务。针对无参考答案的主观任务，我们设计了一套新的评估方法，以替代传统的API模型评分机制。我们对13个先进LLM进行了系统评估，结果表明：相较于其他评估工具，我们提出的评估方法具有更高的相关系数与更强的区分度。此外，我们还探讨了模型参数量、评估维度及归一化方法对评估结果的影响。我们期望F-Eval能够推动LLM基础能力研究的深入发展。