Large language models (LLMs) trained on massive corpora demonstrate impressive capabilities in a wide range of tasks. While there are ongoing efforts to adapt these models to languages beyond English, the attention given to their evaluation methodologies remains limited. Current multilingual benchmarks often rely on back translations or re-implementations of English tests, limiting their capacity to capture unique cultural and linguistic nuances. To bridge this gap for the Korean language, we introduce the HAE-RAE Bench, a dataset curated to challenge models lacking Korean cultural and contextual depth. The dataset encompasses six downstream tasks across four domains: vocabulary, history, general knowledge, and reading comprehension. Unlike traditional evaluation suites focused on token and sequence classification or mathematical and logical reasoning, the HAE-RAE Bench emphasizes a model's aptitude for recalling Korean-specific knowledge and cultural contexts. Comparative analysis with prior Korean benchmarks indicates that the HAE-RAE Bench presents a greater challenge to non-Korean models by disturbing abilities and knowledge learned from English being transferred.

翻译：大规模语料训练的大型语言模型在各类任务中展现出卓越能力。尽管当前有诸多工作致力于将这些模型适配至英语以外的语言，但其评估方法的研究仍显不足。现有的多语言基准测试常依赖回译或对英语测试的复现，难以捕捉独特的文化及语言细微差异。为弥合韩语领域的这一空白，我们提出HAE-RAE Bench数据集，该数据集专门用于挑战缺乏韩国文化及语境深度的模型。该数据集涵盖四个领域的六项下游任务：词汇、历史、常识和阅读理解。与侧重标记分类、序列分类或数学逻辑推理的传统评估套件不同，HAE-RAE Bench聚焦于模型回忆韩语特有知识与文化语境的能力。与先前韩语基准的对比分析表明，HAE-RAE Bench通过干扰从英语迁移而来的能力与知识，对非韩语模型构成了更大的挑战。