The rapid development of large language models (LLMs) has shown promising practical results. However, their low interpretability often leads to errors in unforeseen circumstances, limiting their utility. Many works have focused on creating comprehensive evaluation systems, but previous benchmarks have primarily assessed problem-solving abilities while neglecting the response's uncertainty, which may result in unreliability. Recent methods for measuring LLM reliability are resource-intensive and unable to test black-box models. To address this, we propose UBENCH, a comprehensive benchmark for evaluating LLM reliability. UBENCH includes 3,978 multiple-choice questions covering knowledge, language, understanding, and reasoning abilities. Experimental results show that UBENCH has achieved state-of-the-art performance, while its single-sampling method significantly saves computational resources compared to baseline methods that require multiple samplings. Additionally, based on UBENCH, we evaluate the reliability of 15 popular LLMs, finding GLM4 to be the most outstanding, closely followed by GPT-4. We also explore the impact of Chain-of-Thought prompts, role-playing prompts, option order, and temperature on LLM reliability, analyzing the varying effects on different LLMs.

翻译：大型语言模型（LLMs）的快速发展已展现出具有前景的实际应用成果。然而，其较低的可解释性常在未预见情境下导致错误，从而限制了其实用性。已有诸多研究致力于构建全面的评估体系，但以往的基准测试主要关注问题解决能力，而忽视了响应的不确定性——这可能导致模型不可靠。近期衡量LLM可靠性的方法通常资源消耗大，且无法测试黑盒模型。为此，我们提出了UBENCH，一个用于评估LLM可靠性的综合基准。UBENCH包含3,978道多项选择题，涵盖知识、语言、理解与推理能力。实验结果表明，UBENCH取得了最先进的性能，同时其单次采样方法相比需要多次采样的基线方法显著节省了计算资源。此外，基于UBENCH，我们评估了15个主流LLM的可靠性，发现GLM4表现最为突出，GPT-4紧随其后。我们还探究了思维链提示、角色扮演提示、选项顺序及温度参数对LLM可靠性的影响，并分析了其对不同LLM的差异化作用。