This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to a system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30,000 transcripts of system-simulator interactions based on well-established CS datasets.

翻译：本研究旨在探索在混合主动对话式搜索系统中评估用户反馈的各种方法。尽管对话式搜索系统在多个方面取得了显著进展，但近期研究未能成功整合用户反馈。主要原因是缺乏系统与用户之间的对话交互数据。为此，我们提出了一种基于用户模拟器的框架，用于与多种混合主动对话式搜索系统进行多轮交互。具体而言，我们开发了一个名为ConvSim的用户模拟器，该模拟器在初始化信息需求描述后，能够对系统的响应提供反馈，并回答潜在的澄清问题。我们在多种最先进的段落检索和神经重排序模型上进行的实验表明，有效利用用户反馈可使nDCG@3指标下的检索性能提升16%。此外，我们观察到随着反馈轮次的增加，性能持续提升（三轮后nDCG@3相对提升35%）。这揭示了特定反馈处理模块开发中的研究空白，并为对话式搜索的重大进展开辟了潜力。为支持该主题的进一步研究，我们基于成熟的对话式搜索数据集发布了超过30,000条系统-模拟器交互记录。