Retrieval Augmented Generation (RAG) has gained popularity as a method for conveniently incorporating novel facts that were not seen during the pre-training stage in Large Language Model (LLM)-based Natural Language Generation (NLG) systems. However, LLMs are known to encode significant levels of unfair social biases. The modulation of these biases by RAG in NLG systems is not well understood. In this paper, we systematically study the relationship between the different components of a RAG system and the social biases presented in the text generated across three languages (i.e. English, Japanese and Chinese) and four social bias types (i.e. gender, race, age and religion). Specifically, using the Bias Question Answering (BBQ) benchmark datasets, we evaluate the social biases in RAG responses from document collections with varying levels of stereotypical biases, employing multiple LLMs used as generators. We find that the biases in document collections are often amplified in the generated responses, even when the generating LLM exhibits a low-level of bias. Our findings raise concerns about the use of RAG as a technique for injecting novel facts into NLG systems and call for careful evaluation of potential social biases in RAG applications before their real-world deployment.

翻译：检索增强生成（RAG）作为一种便捷方法，在基于大语言模型（LLM）的自然语言生成（NLG）系统中整合预训练阶段未见的新事实，已获得广泛关注。然而，已知LLM编码了显著程度的不公平社会偏见。RAG在NLG系统中对这些偏见的调节作用尚未得到充分理解。本文系统研究了RAG系统的不同组件与生成文本中呈现的社会偏见之间的关系，涵盖三种语言（即英语、日语和中文）和四种社会偏见类型（即性别、种族、年龄和宗教）。具体而言，我们使用偏见问答（BBQ）基准数据集，通过具有不同刻板偏见程度的文档集合，评估了由多个作为生成器的LLM产生的RAG响应中的社会偏见。研究发现，即使生成LLM本身表现出较低水平的偏见，文档集合中的偏见也常在生成响应中被放大。我们的研究结果对使用RAG作为向NLG系统注入新事实的技术提出了警示，并呼吁在RAG应用实际部署前，对其潜在社会偏见进行审慎评估。