The proliferation of Large Language Models (LLMs) has led to an influx of AI-generated content (AIGC) on the internet, transforming the corpus of Information Retrieval (IR) systems from solely human-written to a coexistence with LLM-generated content. The impact of this surge in AIGC on IR systems remains an open question, with the primary challenge being the lack of a dedicated benchmark for researchers. In this paper, we introduce Cocktail, a comprehensive benchmark tailored for evaluating IR models in this mixed-sourced data landscape of the LLM era. Cocktail consists of 16 diverse datasets with mixed human-written and LLM-generated corpora across various text retrieval tasks and domains. Additionally, to avoid the potential bias from previously included dataset information in LLMs, we also introduce an up-to-date dataset, named NQ-UTD, with queries derived from recent events. Through conducting over 1,000 experiments to assess state-of-the-art retrieval models against the benchmarked datasets in Cocktail, we uncover a clear trade-off between ranking performance and source bias in neural retrieval models, highlighting the necessity for a balanced approach in designing future IR systems. We hope Cocktail can serve as a foundational resource for IR research in the LLM era, with all data and code publicly available at \url{https://github.com/KID-22/Cocktail}.

翻译：随着大语言模型的激增，互联网上涌现了大量人工智能生成内容，使得信息检索系统的语料库从纯人工撰写转变为与LLM生成内容共存。这种AIGC激增对IR系统的影响仍是一个悬而未决的问题，主要挑战在于缺乏面向研究者的专用基准。本文提出Cocktail——一个专为评估LLM时代混合来源数据场景中IR模型而设计的综合性基准。Cocktail包含16个多样化数据集，涵盖不同文本检索任务和领域，其语料库混合了人工撰写与LLM生成内容。此外，为避免LLM中已包含数据集信息可能导致的偏差，我们还引入了一个基于近期事件构建查询的最新数据集NQ-UTD。通过在Cocktail基准数据集上对前沿检索模型进行超1000次实验评估，我们揭示了神经检索模型在排序性能与来源偏差间存在的明显权衡，这凸显了未来IR系统设计中采取平衡方法的必要性。我们希望Cocktail能成为LLM时代IR研究的基础资源，所有数据与代码已公开于\url{https://github.com/KID-22/Cocktail}。