The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

翻译：ChatGPT（一种由大语言模型驱动的聊天机器人）的出现引发了对其传统搜索引擎潜在影响的探讨。本研究通过随机在线实验，将参与者分为两组——分别使用类似ChatGPT的工具和类似谷歌搜索的工具，考察用户在信息检索任务中采用搜索引擎与聊天机器人工具的行为差异。研究结果表明：ChatGPT组在所有任务中耗时更短，但两组整体任务表现无显著差异。值得注意的是，ChatGPT能均衡不同教育水平用户的搜索表现，在回答简单问题和提供通用解决方案方面表现优异，但在事实核查任务中存在不足。用户认为ChatGPT的回复信息质量高于谷歌搜索，但对两者的信任程度相似。此外，使用ChatGPT的参与者在有用性、愉悦性和满意度方面报告了显著更优的用户体验，而两款工具在感知易用性方面无显著差异。然而，ChatGPT也可能导致过度依赖，并生成或复制错误信息，产生不一致的结果。本研究为搜索引擎管理提供了重要启示，并揭示了将聊天机器人技术整合至搜索引擎设计的潜在机遇。