Recently, the emergence of large language models (LLMs) has revolutionized the paradigm of information retrieval (IR) applications, especially in web search. With their remarkable capabilities in generating human-like texts, LLMs have created enormous texts on the Internet. As a result, IR systems in the LLMs era are facing a new challenge: the indexed documents now are not only written by human beings but also automatically generated by the LLMs. How these LLM-generated documents influence the IR systems is a pressing and still unexplored question. In this work, we conduct a quantitative evaluation of different IR models in scenarios where both human-written and LLM-generated texts are involved. Surprisingly, our findings indicate that neural retrieval models tend to rank LLM-generated documents higher. We refer to this category of biases in neural retrieval models towards the LLM-generated text as the \textbf{source bias}. Moreover, we discover that this bias is not confined to the first-stage neural retrievers, but extends to the second-stage neural re-rankers. Then, we provide an in-depth analysis from the perspective of text compression and observe that neural models can better understand the semantic information of LLM-generated text, which is further substantiated by our theoretical analysis. To mitigate the source bias, we also propose a plug-and-play debiased constraint for the optimization objective, and experimental results show the effectiveness. Finally, we discuss the potential severe concerns stemming from the observed source bias and hope our findings can serve as a critical wake-up call to the IR community and beyond. To facilitate future explorations of IR in the LLM era, the constructed two new benchmarks and codes will later be available at \url{https://github.com/KID-22/LLM4IR-Bias}.

翻译：近期，大语言模型（LLMs）的兴起彻底改变了信息检索（IR）应用的范式，尤其是在网络搜索领域。凭借其生成类人文本的卓越能力，LLMs在互联网上创造了海量文本。因此，LLM时代的IR系统面临新挑战：索引文档不仅由人类撰写，还由LLMs自动生成。这些LLM生成的文档如何影响IR系统是一个紧迫且尚未探索的问题。本研究对涉及人类撰写与LLM生成文本的多种IR模型进行了定量评估。令人惊讶的是，我们的发现表明神经检索模型倾向于将LLM生成的文档排名更靠前。我们将神经检索模型中这种对LLM生成文本的偏差称为**源偏差**。此外，我们发现这种偏差不仅限于第一阶神经检索器，还扩展至第二阶神经重排序器。随后，我们从文本压缩角度进行深入分析，观察到神经模型能更好地理解LLM生成文本的语义信息，这一现象通过理论分析得到进一步证实。为缓解源偏差，我们还提出了可插拔的去偏约束优化目标，实验结果表明其有效性。最后，我们讨论了观察到的源偏差可能引发的严重问题，并希望我们的发现能为IR领域及更广泛的社区敲响警钟。为促进LLM时代IR的未来探索，构建的两个新基准和代码将在\url{https://github.com/KID-22/LLM4IR-Bias}上提供。