This research aims to explore various methods for assessing user feedback in mixed-initiative conversational search (CS) systems. While CS systems enjoy profuse advancements across multiple aspects, recent research fails to successfully incorporate feedback from the users. One of the main reasons for that is the lack of system-user conversational interaction data. To this end, we propose a user simulator-based framework for multi-turn interactions with a variety of mixed-initiative CS systems. Specifically, we develop a user simulator, dubbed ConvSim, that, once initialized with an information need description, is capable of providing feedback to a system's responses, as well as answering potential clarifying questions. Our experiments on a wide variety of state-of-the-art passage retrieval and neural re-ranking models show that effective utilization of user feedback can lead to 16% retrieval performance increase in terms of nDCG@3. Moreover, we observe consistent improvements as the number of feedback rounds increases (35% relative improvement in terms of nDCG@3 after three rounds). This points to a research gap in the development of specific feedback processing modules and opens a potential for significant advancements in CS. To support further research in the topic, we release over 30,000 transcripts of system-simulator interactions based on well-established CS datasets.

翻译：本研究旨在探索在混合主动对话式搜索（CS）系统中评估用户反馈的各种方法。尽管CS系统在多个方面取得了显著进展，但近期研究未能成功整合用户反馈。其主要原因之一是缺乏系统与用户之间的对话交互数据。为此，我们提出一种基于用户模拟器的框架，用于与多种混合主动CS系统进行多轮交互。具体而言，我们开发了一个名为ConvSim的用户模拟器，该模拟器在初始化信息需求描述后，能够对系统的响应提供反馈，并回答潜在的澄清性问题。我们在多种最先进的段落检索和神经重排序模型上的实验表明，有效利用用户反馈可使检索性能在nDCG@3指标上提升16%。此外，我们观察到随着反馈轮次增加，性能持续改善（三轮后nDCG@3相对提升35%）。这表明特定反馈处理模块的开发存在研究空白，并为CS领域的重大进展开辟了可能性。为支持该主题的进一步研究，我们基于成熟的CS数据集发布了超过30,000条系统-模拟器交互记录。