Conversational search has seen increased recent attention in both the IR and NLP communities. It seeks to clarify and solve users' search needs through multi-turn natural language interactions. However, most existing systems are trained and demonstrated with recorded or artificial conversation logs. Eventually, conversational search systems should be trained, evaluated, and deployed in an open-ended setting with unseen conversation trajectories. A key challenge is that training and evaluating such systems both require a human-in-the-loop, which is expensive and does not scale. One strategy is to simulate users, thereby reducing the scaling costs. However, current user simulators are either limited to only responding to yes-no questions from the conversational search system or unable to produce high-quality responses in general. In this paper, we show that existing user simulation systems could be significantly improved by a smaller finetuned natural language generation model. However, rather than merely reporting it as the new state-of-the-art, we consider it a strong baseline and present an in-depth investigation of simulating user response for conversational search. Our goal is to supplement existing work with an insightful hand-analysis of unsolved challenges by the baseline and propose our solutions. The challenges we identified include (1) a blind spot that is difficult to learn, and (2) a specific type of misevaluation in the standard setup. We propose a new generation system to effectively cover the training blind spot and suggest a new evaluation setup to avoid misevaluation. Our proposed system leads to significant improvements over existing systems and large language models such as GPT-4. Additionally, our analysis provides insights into the nature of user simulation to facilitate future work.

翻译：对话式搜索近年来在信息检索和自然语言处理领域受到越来越多的关注。它旨在通过多轮自然语言交互来澄清并解决用户的搜索需求。然而，现有大多数系统是利用记录或人工生成的对话日志进行训练和演示的。最终，对话式搜索系统应在开放环境下（面临未知对话轨迹）完成训练、评估与部署。一个关键挑战在于：训练和评估此类系统都需要人在环参与，这不仅成本高昂，而且缺乏可扩展性。一种策略是模拟用户行为，从而降低规模化成本。然而，当前的用户模拟器要么仅限于回应对话式搜索系统的是非问句，要么整体上无法生成高质量响应。在本文中，我们证明通过一个较小的微调自然语言生成模型，现有用户模拟系统可得到显著改进。但我们并非仅将其作为新的最先进技术进行报告，而是将其视为强基线，并深入探究对话式搜索中用户响应的模拟问题。我们的目标是：通过对手工分析基线模型尚未解决的挑战，补充现有工作，并提出解决方案。我们识别出的挑战包括：（1）难以学习的盲区，以及（2）标准设置下特定类型的误评估。我们提出一种新的生成系统以有效覆盖训练盲区，并建议一种新的评估设置以避免误评估。所提系统相比现有系统及GPT-4等大型语言模型均有显著改进。此外，我们的分析揭示了用户模拟的本质特征，可为未来研究提供启示。