Large Language Models (LLMs) trained on massive corpora demonstrate impressive capabilities in a wide range of tasks. While there are ongoing efforts to adapt these models to languages beyond English, the attention given to their evaluation methodologies remains limited. Current multilingual benchmarks often rely on back translations or re-implementations of English tests, limiting their capacity to capture unique cultural and linguistic nuances. To bridge this gap for the Korean language, we introduce HAE-RAE Bench, a dataset curated to challenge models lacking Korean cultural and contextual depth. The dataset encompasses six downstream tasks across four domains: vocabulary, history, general knowledge, and reading comprehension. Contrary to traditional evaluation suites focused on token or sequence classification and specific mathematical or logical reasoning, HAE-RAE Bench emphasizes a model's aptitude for recalling Korean-specific knowledge and cultural contexts. Comparative analysis with prior Korean benchmarks indicates that the HAE-RAE Bench presents a greater challenge to non-native models, by disturbing abilities and knowledge learned from English being transferred.

翻译：大型语言模型（LLMs）在广泛任务中展现出卓越能力，尽管当前正持续推动这些模型适配英语之外的语言，但其评估方法学受到的关注仍然有限。现有多语言基准测试常依赖英语测试的回译或复现，限制了对独特文化及语言细微差异的捕捉能力。为填补韩语领域的这一空白，我们提出HAE-RAE Bench数据集，旨在挑战缺乏韩国文化及语境深度的模型。该数据集涵盖词汇、历史、常识及阅读理解四个领域的六项下游任务。与传统聚焦于词元或序列分类、特定数学或逻辑推理的评估套件不同，HAE-RAE Bench强调模型回忆韩国特定知识与文化语境的能力。与以往韩语基准的对比分析表明，HAE-RAE Benchmark通过干扰从英语习得的能力与知识迁移过程，对非原生模型构成了更大挑战。