The proliferation of open-source Large Language Models (LLMs) from various institutions has highlighted the urgent need for comprehensive evaluation methods. However, current evaluation platforms, such as the widely recognized HuggingFace open LLM leaderboard, neglect a crucial aspect -- uncertainty, which is vital for thoroughly assessing LLMs. To bridge this gap, we introduce a new benchmarking approach for LLMs that integrates uncertainty quantification. Our examination involves eight LLMs (LLM series) spanning five representative natural language processing tasks. Our findings reveal that: I) LLMs with higher accuracy may exhibit lower certainty; II) Larger-scale LLMs may display greater uncertainty compared to their smaller counterparts; and III) Instruction-finetuning tends to increase the uncertainty of LLMs. These results underscore the significance of incorporating uncertainty in the evaluation of LLMs.

翻译：开源大语言模型（LLM）在各机构中的广泛部署凸显了对综合评估方法的迫切需求。然而，当前评估平台（如广受认可的HuggingFace开放LLM排行榜）忽略了关键维度——不确定性，而这对于全面评估LLM至关重要。为弥补这一空白，我们提出了一种整合不确定性量化的新型LLM基准测试方法。我们选取了涵盖五项代表性自然语言处理任务的八个LLM（及LLM系列）进行实验。研究结果表明：I）准确率更高的LLM可能表现出更低的不确定性；II）相较于小型模型，大规模LLM可能展现出更高的不确定性；III）指令微调往往会增加LLM的不确定性。这些结果揭示了将不确定性纳入LLM评估体系的重要性。