The Bias Benchmark for Question Answering (BBQ) is designed to evaluate social biases of language models (LMs), but it is not simple to adapt this benchmark to cultural contexts other than the US because social biases depend heavily on the cultural context. In this paper, we present KoBBQ, a Korean bias benchmark dataset, and we propose a general framework that addresses considerations for cultural adaptation of a dataset. Our framework includes partitioning the BBQ dataset into three classes--Simply-Transferred (can be used directly after cultural translation), Target-Modified (requires localization in target groups), and Sample-Removed (does not fit Korean culture)-- and adding four new categories of bias specific to Korean culture. We conduct a large-scale survey to collect and validate the social biases and the targets of the biases that reflect the stereotypes in Korean culture. The resulting KoBBQ dataset comprises 268 templates and 76,048 samples across 12 categories of social bias. We use KoBBQ to measure the accuracy and bias scores of several state-of-the-art multilingual LMs. The results clearly show differences in the bias of LMs as measured by KoBBQ and a machine-translated version of BBQ, demonstrating the need for and utility of a well-constructed, culturally-aware social bias benchmark.

翻译：问答偏见基准数据集（BBQ）旨在评估语言模型的社会偏见，但由于社会偏见严重依赖文化背景，该基准难以直接适配美国以外的文化语境。本文提出韩语偏见基准数据集KoBBQ，并构建了一个通用框架，系统解决数据集文化适配中的关键问题。该框架将BBQ数据集划分为三类：直接迁移类（经文化翻译后可直接使用）、目标修正类（需对目标群体进行本地化处理）与样本剔除类（不适用于韩国文化场景），同时新增四个韩国文化特有的偏见类别。我们通过大规模问卷调查收集并验证了反映韩国文化刻板印象的社会偏见及其目标群体。最终构建的KoBBQ数据集包含268个模板和76,048个样本，覆盖12个社会偏见类别。利用KoBBQ对多个先进多语言语言模型进行准确率与偏见评分评估，结果显示KoBBQ与机器翻译版BBQ测得的模型偏见存在显著差异，充分证明了构建高质量、文化感知型社会偏见基准的必要性与实用价值。