The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of functionality and security. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs' complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems' integration into society.

翻译：随着具有新兴能力的大型语言模型（LLMs）迅速普及，公众出于好奇开始评测和比较不同LLMs，这促使众多研究者提出了各自的LLM基准。注意到这些基准存在初步缺陷，我们开展了一项研究，采用新颖的统一评估框架，从人员、流程和技术三个维度，并基于功能性与安全性两大支柱，对23个前沿LLM基准进行了批判性评估。我们的研究揭示了显著局限，包括偏差、难以衡量真实推理能力、适应性不足、实施不一致、提示工程复杂性、评估者多样性缺失，以及在综合评估中忽视文化与意识形态规范。讨论部分强调，随着人工智能（AI）的发展，亟需标准化方法论、监管确定性及伦理准则，并倡导从静态基准向动态行为画像的演变，以准确捕捉LLMs的复杂行为与潜在风险。本研究凸显了LLM评估方法范式转变的必要性，强调了通过合作开发公认基准、促进AI系统融入社会的重要性。