Mutation testing is a technique aimed at assessing the effectiveness of test suites by seeding artificial faults into programs. Although available for many platforms and languages, no mutation testing tool is currently available for conversational chatbots, which represent an increasingly popular solution to design systems that can interact with users through a natural language interface. Note that since conversations must be explicitly engineered by the developers of conversational chatbots, these systems are exposed to specific types of faults not supported by existing mutation testing tools. In this paper, we present MutaBot, a mutation testing tool for conversational chatbots. MutaBot addresses mutations at multiple levels, including conversational flows, intents, and contexts. We designed the tool to potentially target multiple platforms, while we implemented initial support for Google Dialogflow chatbots. We assessed the tool with three Dialogflow chatbots and test cases generated with Botium, revealing weaknesses in the test suites.

翻译：变异测试是一种通过向程序中注入人工缺陷来评估测试套件有效性的技术。尽管该技术已适用于多种平台和语言，但目前尚无针对对话式聊天机器人的变异测试工具。对话式聊天机器人作为一种通过自然语言接口与用户交互的系统设计方案，正日益流行。需注意，由于对话必须由开发者显式设计，此类系统易受现有变异测试工具无法支持的特定类型缺陷影响。本文提出MutaBot——一种面向对话式聊天机器人的变异测试工具。MutaBot在对话流程、意图和上下文等多个层面实施变异。我们设计的工具可潜在支持多平台，并初步实现了对Google Dialogflow聊天机器人的支持。通过使用Botium生成的测试用例对三个Dialogflow聊天机器人进行评估，我们揭示了测试套件中存在的缺陷。