As large language models (LLMs) are more frequently used in retrieval-augmented generation pipelines, it is increasingly relevant to study their behavior under knowledge conflicts. Thus far, the role of the source of the retrieved information has gone unexamined. We address this gap with a novel framework to investigate how source preferences affect LLM resolution of inter-context knowledge conflicts in English, motivated by interdisciplinary research on credibility. With a comprehensive, tightly-controlled evaluation of 13 open-weight LLMs, we find that LLMs prefer institutionally-corroborated information (e.g., government or newspaper sources) over information from people and social media. However, these source preferences can be reversed by simply repeating information from less credible sources. To mitigate repetition effects and maintain consistent preferences, we propose a novel method that reduces repetition bias by up to 99.8%, while also maintaining at least 88.8% of original preferences. We release all data and code to encourage future work on credibility and source preferences in knowledge-intensive NLP.

翻译：随着大型语言模型（LLM）在检索增强生成流程中的应用日益频繁，研究其在知识冲突下的行为变得愈发重要。迄今为止，检索信息来源的作用尚未得到充分考察。受跨学科可信度研究的启发，我们提出一个新颖框架来探究信源偏好如何影响LLM在英语语境中解决跨上下文知识冲突。通过对13个开源权重LLM进行全面且严格控制的评估，我们发现LLM更倾向于机构验证的信息（如政府或新闻媒体来源），而非来自个人和社交媒体的信息。然而，仅通过重复低可信度来源的信息即可逆转这些信源偏好。为减轻重复效应并保持偏好一致性，我们提出一种新方法，可将重复偏差降低高达99.8%，同时保持至少88.8%的原始偏好。我们公开所有数据与代码，以促进知识密集型自然语言处理领域关于可信度与信源偏好的后续研究。