Conversational search has seen increased recent attention in both the IR and NLP communities. It seeks to clarify and solve a user's search need through multi-turn natural language interactions. However, most existing systems are trained and demonstrated with recorded or artificial conversation logs. Eventually, conversational search systems should be trained, evaluated, and deployed in an open-ended setting with unseen conversation trajectories. A key challenge is that training and evaluating such systems both require a human-in-the-loop, which is expensive and does not scale. One strategy for this is to simulate users, thereby reducing the scaling costs. However, current user simulators are either limited to only respond to yes-no questions from the conversational search system, or unable to produce high quality responses in general. In this paper, we show that current state-of-the-art user simulation system could be significantly improved by replacing it with a smaller but advanced natural language generation model. But rather than merely reporting this new state-of-the-art, we present an in-depth investigation of the task of simulating user response for conversational search. Our goal is to supplement existing works with an insightful hand-analysis of what challenges are still unsolved by the advanced model, as well as to propose our solutions for them. The challenges we identified include (1) dataset noise, (2) a blind spot that is difficult for existing models to learn, and (3) a specific type of misevaluation in the standard empirical setup. Except for the dataset noise issue, we propose solutions to cover the training blind spot and to avoid the misevaluation. Our proposed solutions lead to further improvements. Our best system improves the previous state-of-the-art significantly.

翻译：对话式搜索近年来在信息检索和自然语言处理领域受到越来越多的关注。它旨在通过多轮自然语言交互来明确并解决用户的搜索需求。然而，现有的大多数系统都是使用记录或人工合成的对话日志进行训练和演示的。最终，对话式搜索系统应在开放环境下，使用未见过的对话轨迹进行训练、评估和部署。一个关键挑战是，训练和评估此类系统都需要人类参与，这既昂贵又难以扩展。解决这一问题的策略之一是模拟用户，从而降低扩展成本。然而，当前的用户模拟器要么仅限于回应对话式搜索系统的“是/否”问题，要么总体上无法生成高质量的回答。在本文中，我们表明，当前最先进的用户模拟系统可以通过替换为更小但更先进的自然语言生成模型来显著改进。但本文并非仅仅报告这一新的最先进水平，而是对模拟对话式搜索用户响应这一任务进行深入研究。我们的目标是通过对先进模型仍未解决的挑战进行深入的手动分析，来补充现有研究，并提出相应的解决方案。我们识别出的挑战包括：（1）数据集噪声，（2）现有模型难以学习的盲点，以及（3）标准实验设置中的一种特定类型的误评估。除了数据集噪声问题外，我们提出了覆盖训练盲点和避免误评估的解决方案。我们提出的方案带来了进一步的改进。我们的最佳系统显著提升了之前的最先进水平。