The current evaluation of Large Language Models (LLMs) predominantly relies on benchmarks focusing on their embedded knowledge by testing through multiple-choice questions (MCQs), a format inherently suited for automated evaluation. Our study extends this evaluation to explore LLMs' pragmatic competence--a facet previously underexamined before the advent of sophisticated LLMs, specifically in the context of Korean. We employ two distinct evaluation setups: the conventional MCQ format, adapted for automatic evaluation, and Open-Ended Questions (OEQs), assessed by human experts, to examine LLMs' narrative response capabilities without predefined options. Our findings reveal that GPT-4 excels, scoring 81.11 and 85.69 in the MCQ and OEQ setups, respectively, with HyperCLOVA X, an LLM optimized for Korean, closely following, especially in the OEQ setup, demonstrating a score of 81.56 with a marginal difference of 4.13 points compared to GPT-4. Furthermore, while few-shot learning strategies generally enhance LLM performance, Chain-of-Thought (CoT) prompting introduces a bias toward literal interpretations, hindering accurate pragmatic inference. Considering the growing expectation for LLMs to understand and produce language that aligns with human communicative norms, our findings emphasize the importance for advancing LLMs' abilities to grasp and convey sophisticated meanings beyond mere literal interpretations.

翻译：当前对大型语言模型（LLMs）的评估主要依赖以选择题（MCQs）形式测试其内在知识的基准测试，这种格式天然适用于自动化评估。本研究将评估范围拓展至探索LLMs的语用能力——这在复杂LLMs出现之前是一个尚未充分研究的维度，尤其针对韩语语境。我们采用两种不同的评估设置：传统选择题格式（适用于自动评估）和开放式问答题（OEQs）（由人类专家评估），以检验LLMs在无预设选项情况下的叙事回应能力。研究结果表明，GPT-4表现优异，在选择题和开放式问答题设置中分别获得81.11分和85.69分；而针对韩语优化的LLM HyperCLOVA X紧随其后，尤其在开放式问答题设置中表现突出，以81.56分与GPT-4仅相差4.13分。此外，虽然小样本学习策略通常能提升LLM性能，但思维链（CoT）提示会引入字面解释偏差，阻碍准确的语用推理。考虑到对LLMs理解并生成符合人类交际规范语言的期望日益增长，我们的发现强调了提升LLMs超越字面解释、掌握并传达复杂含义能力的重要性。