Warning: This paper consists of examples representing regional biases in Indian regions that might be offensive towards a particular region. While social biases corresponding to gender, race, socio-economic conditions, etc., have been extensively studied in the major applications of Natural Language Processing (NLP), biases corresponding to regions have garnered less attention. This is mainly because of (i) difficulty in the extraction of regional bias datasets, (ii) disagreements in annotation due to inherent human biases, and (iii) regional biases being studied in combination with other types of social biases and often being under-represented. This paper focuses on creating a dataset IndRegBias, consisting of regional biases in an Indian context reflected in users' comments on popular social media platforms, namely Reddit and YouTube. We carefully selected 25,000 comments appearing on various threads in Reddit and videos on YouTube discussing trending topics on regional issues in India. Furthermore, we propose a multilevel annotation strategy to annotate the comments describing the severity of regional biased statements. To detect the presence of regional bias and its severity in IndRegBias, we evaluate open-source Large Language Models (LLMs) and Indic Language Models (ILMs) using zero-shot, few-shot, and fine-tuning strategies. We observe that zero-shot and few-shot approaches show lower accuracy in detecting regional biases and severity in the majority of the LLMs and ILMs. However, the fine-tuning approach significantly enhances the performance of the LLM in detecting Indian regional bias along with its severity.

翻译：警告：本文包含代表印度地区区域偏见的示例，可能对特定地区具有冒犯性。尽管在自然语言处理（NLP）的主要应用中，与性别、种族、社会经济状况等相对应的社会偏见已得到广泛研究，但与区域相对应的偏见却较少受到关注。这主要是因为：（i）区域偏见数据集的提取困难；（ii）由于人类固有偏见导致的标注不一致；以及（iii）区域偏见常与其他类型的社会偏见结合研究，且往往代表性不足。本文专注于创建数据集IndRegBias，该数据集包含印度语境下的区域偏见，这些偏见反映在用户于流行社交媒体平台（即Reddit和YouTube）上的评论中。我们精心筛选了Reddit上各类主题帖及YouTube上讨论印度区域热点话题的视频中出现的25,000条评论。此外，我们提出了一种多级标注策略，用于标注描述区域偏见陈述严重程度的评论。为了检测IndRegBias中区域偏见的存在及其严重程度，我们使用零样本、少样本和微调策略评估了开源大语言模型（LLMs）及印度语言模型（ILMs）。我们观察到，在大多数LLMs和ILMs中，零样本和少样本方法在检测区域偏见及其严重程度方面准确率较低。然而，微调方法显著提升了LLM在检测印度区域偏见及其严重程度方面的性能。