For Large Language Models (LLMs) to be effectively deployed in a specific country, they must possess an understanding of the nation's culture and basic knowledge. To this end, we introduce National Alignment, which measures an alignment between an LLM and a targeted country from two aspects: social value alignment and common knowledge alignment. Social value alignment evaluates how well the model understands nation-specific social values, while common knowledge alignment examines how well the model captures basic knowledge related to the nation. We constructed KorNAT, the first benchmark that measures national alignment with South Korea. For the social value dataset, we obtained ground truth labels from a large-scale survey involving 6,174 unique Korean participants. For the common knowledge dataset, we constructed samples based on Korean textbooks and GED reference materials. KorNAT contains 4K and 6K multiple-choice questions for social value and common knowledge, respectively. Our dataset creation process is meticulously designed and based on statistical sampling theory and was refined through multiple rounds of human review. The experiment results of seven LLMs reveal that only a few models met our reference score, indicating a potential for further enhancement. KorNAT has received government approval after passing an assessment conducted by a government-affiliated organization dedicated to evaluating dataset quality. Samples and detailed evaluation protocols of our dataset can be found in https://selectstar.ai/ko/papers-national-alignment

翻译：摘要：为使大型语言模型（LLM）在特定国家得到有效部署，它们必须理解该国的文化并掌握基础常识。为此，我们提出“国家对齐”（National Alignment）概念，从两个维度衡量LLM与目标国家之间的对齐程度：社会价值观对齐与常识知识对齐。社会价值观对齐评估模型理解国家特定社会价值观的能力，而常识知识对齐则考察模型掌握与国家相关的基础常识。我们构建了KorNAT，这是首个衡量与韩国国家对齐程度的基准数据集。在社会价值观数据集方面，我们从涵盖6,174名韩国参与者的全国性大规模调查中获取了真实标签；在常识知识数据集方面，我们基于韩国教科书和高中同等学历考试（GED）参考资料构建了样本。KorNAT分别包含4,000道和6,000道社会价值观与常识知识选择题。我们的数据集构建过程经过精心设计，基于统计抽样理论，并通过多轮人工审核进行优化。对七个LLM的实验结果显示，仅少数模型达到我们的参考分数，表明模型性能仍有进一步提升空间。KorNAT已通过由政府下属专门评估数据集质量的机构进行的审核，并获得了政府批准。数据集样本及详细评估协议可访问https://selectstar.ai/ko/papers-national-alignment获取。