The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods. However, current evaluation platforms, such as the widely recognized HuggingFace open LLM leaderboard, neglect a crucial aspect -- uncertainty, which is vital for thoroughly assessing LLMs. To bridge this gap, we introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our examination involves eight LLMs (LLM series) spanning five representative natural language processing tasks. Additionally, we introduce an uncertainty-aware evaluation metric, UAcc, which takes into account both prediction accuracy and prediction uncertainty. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs. By taking uncertainty into account, our new UAcc metric can either amplify or diminish the relative improvement of one LLM over another and may even change the relative ranking of two LLMs. These results underscore the significance of incorporating uncertainty in the evaluation of LLMs.

翻译：随着来自各机构开源大型语言模型（LLMs）的激增，对全面评估方法的需求日益迫切。然而，当前评估平台（如广受认可的HuggingFace开放LLM排行榜）忽略了一个关键方面——不确定性，这对于彻底评估LLMs至关重要。为弥补这一不足，我们提出了一种集成不确定性量化的LLM新型基准评估方法。本研究考察了涵盖五项代表性自然语言处理任务的八种LLM（LLM系列）。此外，我们引入了一种不确定性感知评估指标UAcc，该指标同时考虑了预测准确性与预测不确定性。研究结果表明：I）准确性较高的LLM可能表现出更低的可信度；II）较大规模的LLM相较于其较小版本可能展现出更高的不确定性；III）指令微调倾向于增加LLM的不确定性。通过纳入不确定性考量，新提出的UAcc指标既能放大也能缩小两个LLM之间的相对改进幅度，甚至可能改变两者的相对排名。这些结果凸显了在LLM评估中纳入不确定性的关键意义。