Socio-cultural stereotypical bias is an important consideration in the development and deployment of NLP systems. It is however often considered only at the national level, despite rich subnational socio-cultural structures. We present AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa with its unique historically multicultural setting. It covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions in both English and Devanagari Konkani. We then evaluate stereotypical bias in five multilingual encoder models on this benchmark. We find near-chance scores in Konkani, reflecting language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models. Queried in English, models with a stronger Indian language coverage show higher bias for pan-Indian groups than hyperlocal Goan groups. This suggests the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge. Our findings highlight a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities.

翻译：社会文化刻板偏见是自然语言处理系统开发与部署中的重要考量因素。尽管存在丰富的次国家社会文化结构，此类偏见却往往仅在国家层面被纳入考量。本文提出AmchiBias——首个专门用于衡量印度果阿邦（该地区具有独特的历史多元文化背景）社会文化刻板偏见的基准数据集。该数据集涵盖多个果阿身份群体，包含313个跨八个社会人口维度的最小配对样本（同时提供英语和天城体孔卡尼语版本）。我们在此基础上评估了五种多语言编码器模型中的刻板偏见。研究发现，模型在孔卡尼语上的表现接近随机水平，反映出通用多语言模型的语言能力不足，以及印度语模型缺乏对果阿文化的能力。当使用英语查询时，具有更强印度语言覆盖范围的模型对泛印度群体的偏见程度高于对本地化果阿群体。这表明英语信号反映的是泛印地预训练关联，而非真正的果阿文化知识。我们的研究结果凸显了针对超本地化社区身份的低资源多语言NLP评估中存在的关键缺口。