Generative AI is being increasingly integrated into web search for the convenience it provides users. In this work, we aim to understand how generative AI disrupts web search by retrieving and presenting the information and sources differently from traditional search engines. We introduce a public benchmark dataset of 11,500 user queries to support our study and future research of generative search. We compare the search results returned by Google's search engine, the accompanying AI Overview (AIO), and Gemini Flash 2.5 for each query. We have made several key findings. First, we find that for 51.5\% of representative, real-user queries, AIOs are generated, and are displayed above the organic search results. Controversial questions frequently result in an AIO. Second, we show that the retrieved sources are substantially different for each search engine (<0.2 average Jaccard similarity). Traditional Google search is significantly more likely to retrieve information from popular or institutional websites in government or education, while generative search engines are significantly more likely to retrieve Google-owned content. Third, we observe that websites that block Google's AI crawler are significantly less likely to be retrieved by AIOs, despite having access to the content. Finally, AIOs are less consistent when processing two runs of the same query, and are less robust to minor query edits. Our findings have important implications for understanding how generative search impacts website visibility, the effectiveness of generative engine optimization techniques, and the information users receive. We call for revenue frameworks to foster a sustainable and mutually beneficial ecosystem for publishers and generative search providers.

翻译：生成式人工智能正日益融入网络搜索，为用户带来便利。本研究旨在理解生成式AI如何通过以不同于传统搜索引擎的方式检索并呈现信息与来源，从而颠覆网络搜索。我们引入了一个包含11500条用户查询的公开基准数据集，以支持本研究及未来关于生成式搜索的探索。我们针对每条查询，比较了谷歌搜索引擎、其配套的AI概述（AIO）以及Gemini Flash 2.5返回的搜索结果，并得出若干关键发现。首先，我们发现对于51.5%具代表性的真实用户查询，AI概述被生成并显示在自然搜索结果之上，且争议性问题常触发AI概述。其次，各搜索引擎检索到的来源存在显著差异（平均Jaccard相似度低于0.2）。传统谷歌搜索更倾向于从政府或教育领域的知名或机构网站获取信息，而生成式搜索引擎则更倾向于检索谷歌自有内容。第三，我们观察到，屏蔽谷歌AI爬虫的网站在AI概述中被检索到的概率显著降低，即便其内容仍可访问。最后，AI概述在对同一查询进行两次运行处理时表现不一致，且对细微查询修改的鲁棒性较弱。这些发现对于理解生成式搜索如何影响网站可见性、生成式引擎优化技术的有效性以及用户所接收到的信息具有重要意义。我们呼吁建立收入框架，以促进出版商与生成式搜索提供商之间可持续且互惠互利的生态系统。