This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to a system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30,000 transcripts of system-simulator interactions based on well-established CS datasets.

翻译：本研究旨在探索混合主动对话式搜索（CS）系统中评估用户反馈的多种方法。尽管CS系统在多个方面取得了显著进展，但近期研究未能成功整合用户反馈。主要原因之一是缺乏系统与用户之间的对话交互数据。为此，我们提出了一种基于用户模拟器的框架，用于与多种混合主动CS系统进行多轮交互。具体而言，我们开发了一个名为ConvSim的用户模拟器，该模拟器在通过信息需求描述初始化后，能够对系统的响应提供反馈，并回答潜在的澄清问题。我们在多种最先进的段落检索和神经重排序模型上进行的实验表明，有效利用用户反馈可使nDCG@3指标的检索性能提升16%。此外，我们观察到随着反馈轮次增加，性能持续提升（三轮后nDCG@3相对提升35%）。这表明在特定反馈处理模块的开发方面存在研究空白，并为CS领域的重大进展开辟了可能性。为支持该主题的进一步研究，我们基于成熟的CS数据集发布了超过30,000条系统-模拟器交互记录。