As large language models (LLMs) are more frequently used in retrieval-augmented generation pipelines, it is increasingly relevant to study their behavior under knowledge conflicts. Thus far, the role of the source of the retrieved information has gone unexamined. We address this gap with a novel framework to investigate how source preferences affect LLM resolution of inter-context knowledge conflicts in English, motivated by interdisciplinary research on credibility. With a comprehensive, tightly-controlled evaluation of 13 open-weight LLMs, we find that LLMs prefer institutionally-corroborated information (e.g., government or newspaper sources) over information from people and social media. However, these source preferences can be reversed by simply repeating information from less credible sources. To mitigate repetition effects and maintain consistent preferences, we propose a novel method that reduces repetition bias by up to 99.8%, while also maintaining at least 88.8% of original preferences. We release all data and code to encourage future work on credibility and source preferences in knowledge-intensive NLP.

翻译：随着大语言模型在检索增强生成流程中的使用日益频繁，研究其在知识冲突下的行为变得愈发重要。迄今为止，检索信息的来源角色尚未得到充分检验。受跨学科可信度研究的启发，我们提出一个新颖框架来探究信源偏好如何影响大语言模型处理英语跨语境知识冲突。通过对13个开源权重的大语言模型进行全面且严格控制的评估，我们发现大语言模型更倾向于机构佐证的信息（如政府或新闻媒体来源），而非来自个人和社交媒体的信息。然而，仅需重复来自低可信度信源的信息即可逆转这些信源偏好。为减轻重复效应并保持偏好一致性，我们提出一种新方法，可将重复偏差降低高达99.8%，同时保持至少88.8%的原始偏好。我们公开所有数据与代码，以促进知识密集型自然语言处理领域关于可信度与信源偏好的未来研究。