AI is promising in assisting UX evaluators with analyzing usability tests, but its judgments are typically presented as non-interactive visualizations. Evaluators may have questions about test recordings, but have no way of asking them. Interactive conversational assistants provide a Q&A dynamic that may improve analysis efficiency and evaluator autonomy. To understand the full range of analysis-related questions, we conducted a Wizard-of-Oz design probe study with 20 participants who interacted with simulated AI assistants via text or voice. We found that participants asked for five categories of information: user actions, user mental model, help from the AI assistant, product and task information, and user demographics. Those who used the text assistant asked more questions, but the question lengths were similar. The text assistant was perceived as significantly more efficient, but both were rated equally in satisfaction and trust. We also provide design considerations for future conversational AI assistants for UX evaluation.

翻译：人工智能在辅助用户体验评估者分析可用性测试方面具有前景，但其判断通常以非交互式可视化形式呈现。评估者可能对测试记录存在疑问，却无法直接提问。交互式对话助手提供的问答动态机制或可提升分析效率与评估者的自主性。为全面理解与分析相关的各类问题，我们开展了一项巫师Oz设计探索研究，组织20名参与者通过文本或语音方式与模拟AI助手互动。研究发现，参与者提出了五类信息需求：用户行为、用户心理模型、AI助手提供的帮助、产品与任务信息以及用户人口学特征。使用文本助手的参与者提问数量更多，但问题长度相近。文本助手被认为效率显著更高，但两者在满意度与信任度评分上无显著差异。我们还为未来用于用户体验评估的对话式AI助手提供了设计考量。