Task-oriented dialogue systems (TDSs) are assessed mainly in an offline setting or through human evaluation. The evaluation is often limited to single-turn or is very time-intensive. As an alternative, user simulators that mimic user behavior allow us to consider a broad set of user goals to generate human-like conversations for simulated evaluation. Employing existing user simulators to evaluate TDSs is challenging as user simulators are primarily designed to optimize dialogue policies for TDSs and have limited evaluation capabilities. Moreover, the evaluation of user simulators is an open challenge. In this work, we propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems. We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities. Our user simulator constructs a metaphorical user model that assists the simulator in reasoning by referring to prior knowledge when encountering new items. We estimate the quality of simulators by checking the simulated interactions between simulators and variants. Our experiments are conducted using three TDS datasets. The proposed user simulator demonstrates better consistency with manual evaluation than an agenda-based simulator and a seq2seq model on three datasets; our tester framework demonstrates efficiency and has been tested on multiple tasks, such as conversational recommendation and e-commerce dialogues.

翻译：任务型对话系统（TDS）主要通过离线评估或人工评估进行测试，但此类评估往往局限于单轮对话或耗时较长。作为替代方案，通过模拟用户行为的用户模拟器，能够覆盖更广泛的用户目标，生成类人的对话以进行模拟评估。然而，现有用户模拟器主要用于优化TDS对话策略，其评估能力有限，且用户模拟器本身的评估仍是一大开放挑战。本研究提出一种用于端到端TDS评估的隐喻性用户模拟器——若模拟器能在与系统交互中模拟用户的类比思维，则称其为隐喻性用户模拟器。我们同时提出基于测试器的评估框架，用于生成不同能力的对话系统变体。该用户模拟器构建了隐喻性用户模型，通过参考与新增项目相关的先验知识辅助模拟器进行推理。我们通过检验模拟器与变体之间的模拟交互来评估模拟器质量。基于三个TDS数据集的实验表明，所提用户模拟器与人工评估的一致性优于基于议程的模拟器和序列到序列模型；测试器框架在对话推荐与电子商务对话等多任务中展现出高效性。