Large Language Models (LLMs) have demonstrated remarkable capabilities across various applications, fundamentally reshaping the landscape of natural language processing (NLP) research. However, recent evaluation frameworks often rely on the output probabilities of LLMs for predictions, primarily due to computational constraints, diverging from real-world LLM usage scenarios. While widely employed, the efficacy of these probability-based evaluation strategies remains an open research question. This study aims to scrutinize the validity of such probability-based evaluation methods within the context of using LLMs for Multiple Choice Questions (MCQs), highlighting their inherent limitations. Our empirical investigation reveals that the prevalent probability-based evaluation method inadequately aligns with generation-based prediction. Furthermore, current evaluation frameworks typically assess LLMs through predictive tasks based on output probabilities rather than directly generating responses, owing to computational limitations. We illustrate that these probability-based approaches do not effectively correspond with generative predictions. The outcomes of our study can enhance the understanding of LLM evaluation methodologies and provide insights for future research in this domain.

翻译：大语言模型（LLMs）在各类应用中展现出卓越能力，从根本上重塑了自然语言处理（NLP）研究格局。然而，受计算资源限制，当前评估框架常依赖LLMs的输出概率进行预测，这与真实应用场景存在偏离。尽管此类基于概率的评估策略被广泛使用，但其有效性仍是开放性问题。本研究旨在剖析在LLMs处理多项选择题（MCQs）时，此类基于概率的评估方法的有效性，揭示其固有局限性。实证研究表明，当前主流的基于概率的评估方法与基于生成的预测存在显著不匹配。此外，由于计算限制，现有评估框架通常通过输出概率驱动的预测任务评估LLMs，而非直接采用生成式响应。我们证实这些基于概率的方法未能与生成式预测有效对应。本研究结果可深化对LLM评估方法论的理解，并为该领域未来研究提供启示。