Using a sample of 25,000 Bing Copilot conversations, we study how the agent responds to users of varying levels of domain expertise and the resulting impact on user experience along multiple dimensions. Our findings show that across a variety of topical domains, the agent largely responds at proficient or expert levels of expertise (77% of conversations) which correlates with positive user experience regardless of the user's level of expertise. Misalignment, such that the agent responds at a level of expertise below that of the user, has a negative impact on overall user experience, with the impact more profound for more complex tasks. We also show that users engage more, as measured by the number of words in the conversation, when the agent responds at a level of expertise commensurate with that of the user. Our findings underscore the importance of alignment between user and AI when designing human-centered AI systems, to ensure satisfactory and productive interactions.

翻译：基于25,000次Bing Copilot对话样本，我们研究了智能体如何回应用户的不同领域专业水平，及其对多维用户体验的影响。研究发现：在各类主题领域中，智能体主要以精通或专家级专业水平进行回应（占对话的77%），这与积极的用户体验相关，且与用户自身专业水平无关。当智能体回应专业水平低于用户时，这种错位会对整体用户体验产生负面影响，且在处理更复杂任务时影响更为显著。研究还表明，当智能体回应的专业水平与用户相匹配时，用户参与度更高（以对话字数衡量）。本研究的发现强调了在设计以人为本的AI系统时，确保用户与AI专业水平对齐的重要性，以保障交互的满意度和高效性。